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ABSTRACT 

The  flow  of  messages  in  a message-switched  data  communication  net- 
work is  modeled  in  a continuous  dynamical  state  space.  The  state  vari- 
ables represent  message  storage  at  the  nodes  and  the  control  variables 
represent  message  flow  rates  along  the  links.  A deterministic  linear 
cost  functional  is  defined  which  is  the  weighted  total  message  delay 
in  the  network  when  we  stipulate  that  all  the  message  backlogs  are 
emptied  at  the  final  time  and  the  inputs  are  known.  The  desired  mini- 
mization of  the  cost  functional  results  in  a linear  optimal  control 
problem  with  linear  state  and  control  variable  inequality  constraints. 

The  remainder  of  the  thesis  is  devoted  to  finding  the  feedback 
solution  to  the  optimal  control  problem  when  all  the  inputs  are  con- 
stant in  time.  First,  the  necessary  conditions  of  optimality  are 
derived  and  shown  to  be  sufficient.  The  pointwise  minimization  in  time 
is  a linear  program  and  the  optimal  control  is  seen  to  be  of  the  bang- 
bang  variety.  Utilizing  the  necessary  conditions  it  is  shown  that  the 
feedback  regions  of  interest  are  convex  polyhedral  cones  in  the  state 
space.  A method  is  then  described  for  constructing  these  regions  from 
a comprehensive  set  of  optimal  trajectories  constructed  backward  in 
time  from  the  final  time.  Techniques  in  linear  programming  are  em- 
ployed freely  throughout  the  development,  particularly  the  geometrical 
interpretation  of  linear  programs  and  parametric  linear  programming. 

There  are  several  properties  of  the  method  which  complicate  its 
formulation  as  a compact  algorithm  for  general  network  problems.  How- 
ever, in  the  case  of  problems  involving  networks  with  single  destina- 
tions and  all  unity  weiglit  ings  in  the  cost  functional  it  is  shown  tliat 
these  properties  do  not  apply.  A computer  implementcU'le  algoritlim  is 
tiien  detailed  for  the  construction  of  the  feedback  solution. 


-2a- 


Thesis  Supervisor:  Adrian  Segall 

Title:  Assistant  Professor  Electrical  Engineering 

Thesis  Supervisor:  Wallace  E.  Vander  Velde 

Title:  Professor  of  Aeronautics  and  Astronautics 

Thesis  Supervisor:  Alan  S.  Willsky 

Title:  Associate  Professor  of  Electrical  Engineering 


ACKNOWLEDGEMENTS 


I wish  to  express  my  sincere  gratitude  to  the  members  of  my  Doc- 
toral Committee  for  invaluable  counsel  and  unfailing  personal  support. 
Professor  Adrian  Segall  stimulated  my  interest  in  data  communication 
networks,  suggested  the  topic  of  this  thfiis,  and  generously  contri- 
buted of  his  time  as  colleague  and  friend  during  this  research.  From 
him  I have  learned  that  perserverance  is  the  most  important  part  of 
research.  Professor  Wallace  E.  Vander  Velde,  chairman  of  the  commiLtee, 
has  served  as  mentor  and  teacher  throughout  my  doctoral  program.  His 
wise  advice,  with  regard  to  this  thesis  and  other  matters,  is  greatly 
appreciated.  My  association  with  Professor  Alan  S.  Willsky,  both 
during  thesis  research  and  previously,  has  been  most  rewarding.  His 
intellectual  and  personal  energy,  combined  with  a fine  sense  of  humor, 
provide  an  atmosphere  which  I find  both  stimulating  and  enjoyable. 

My  research  experience  at  the  Electronic  Systems  Laboratory  has 
been  a very  pleasant  and  productive  one.  I offer  my  heartfelt  thanks 
to  those  faculty  members,  staff  members  and  fellow  students  who  have 
helped  make  it  so.  In  particular:  to  Professor  John  M.  Wozencraft,  for 
graciously  providing  the  administrative  support  which  made  this  research 
possible;  to  Professor  Sanjoy  K.  Mitter,  for  serving  as  a thesis  reader 
and  sharing  of  his  expertise  in  optimal  control  theory;  to  Professor 


-3- 


TABLE  OF  CONTENTS 


Page 

ABSTRACT  2 
ACKNOWLEDGEMENTS  3 
TABLE  OF  CONTENTS  ^ 


Chapter  1 

INTRODUCTION 

9 

1.1 

Introduction  to  Data  Communication  Networks 

9 

1.2 

Discussion  of  the  Message  Routing  Problem 

12 

1.3 

Thesis  Overview 

17 

1.3.1  Obiective  of  Thesis 

17 

1.3.2  Approach 

18 

1.3.3  Preview  of  Results 

20 

1.4 

Synopsis  of  Thesis 

23 

Chapter  2 

NEW  DATA  COMMUNICATION  NETWORK  MODEL  AND 
OPTIMAL  CONTROL  PROBLEM  FOR  DYNAMIC  ROUTING 

26 

2.1 

Introduction 

26 

2.2 

Basic  Elements  of  the  State  Space  Model 

28 

2.2.1  Topological  Representation 

28 

2.2.2  Message  Flow  and  Storage  Representation 

29 

2.3 

Dynamic  Equations  and  Constraints 

33 

2.4 

Performance  Index  and  Optimal  Control  Problem 
Statement 

3S 

-5- 


preceding  page  blank  - not  filmed 


-6- 


Page 

2. 5 Discussion  of  the  Optimal  Control  Problem  39 

2.6  Previous  Work  on  Optimal  Control  Problems  with 

State  Variable  Inequality  Constraints  41 

2.7  Open  Loop  Solutions  45 

2.7.1  Linear  Programming  Through  Discretization  45 

2.7.2  Penalty  Function  Method  47 

Chapter  3 tKEDBACK  SOLUTION  TO  THE  OPTIMkT.  CONTPOL 

PROBLEM  WITH  CONSTANT  INPUTS  50 

3 . 1 Introduction  50 

3. 2 Feedback  Solution  Eundamentals  52 

3.2.1  Necessary-  and  Sufficient  Conditions  52 

3.2.2  Controllability  to  Zero  tor  Constant 

Inputs  67 

3.2.3  Geometrical  Characterization  of  tlie 

Feedback  Space  for  Constant  Inputs  72 

3 . 3 Backward  Construction  of  the  Feedback  Space  for 

Constant  Inputs  91 

3.3.1  Introductory  Examples  of  Backward  Boundary 

Sequence  Technique  91 

3.3.2  Technique  for  the  Backward  Construction  of 

the  Feedback  Space  113 

3. 3. 2.1  Preliminaries  113 

3. 3. 2. 2 Description  of  the  Algorithm  115 

3. 3. 2. 3 Discussion  of  the  Algorithm  135 

Basic  Procedures  of  the  Algorithm  142 


3.  3.  3 


-7- 


Page 


3. 3. 3.1 

Solution  of  Constrained 

Optimization  Problem 

142 

3. 3. 3. 2 

Determination  of  Leave-the- 

Boundary  Costates,  Subregions 

and  Global  Optimality 

14  7 

3. 3. 3. 3 

Construction  of  Non-Break 

Feedback  Control  Regions 

168 

3. 3. 3. 4 

Construction  of  Break 

Feedback  Control  Regions 

175 

3.3.4  Discussion  of  Fundamental  Properties  of 

the  Algorithm 

180 

3. 3. 4.1 

Non-Global  Optimality  of 

Certain  Sequences 

181 

3. 3. 4. 2 

Non-Uniqueness  of  Leave-the- 

Boundary  Costates 

186 

3. 3. 4. 3 

Subregions 

190 

3. 3.4.4 

Return  of  States  to  Boundary 

Backward  in  Time 

196 

3.4 

Summary 

202 

Chapter  4 

FEEDBACK  ROUTING 
WEIGHTINGS 

WITH  CONSTANT  INPUTS  AND  UNITY 

204 

4.1 

Introduction 

204 

4.2 

Preliminaries 

206 

4.3 

Special  Properties  of  the  Algorithm 

216 

4.4 

Computational  Algorithm 

267 

4.5 

Extension  to  General  Network  Problems 

278 

Chapter  5 

CONCLUSION 

284 

5.1 

Discussion 

284 

J 


Page 


5.2 

Summary  of  Results  and  Related  Observations 

288 

5.2.1  Results 

288 

5.2.2  Related  Observations 

289 

5.  3 

Contributions 

292 

5.4 

Suggestions  for  Further  Work 

293 

Appendix  A 

. NDING  ALL  EXTREME  POINT  SOLUTIONS  TO  THE 
CONSTRAINED  OPTIMIZATION  PROBLEM 

296 

Appendix  B 

EXA.MPLE  OF  NON-BREAK  AND  BREAK  FEEDBACK  CONTROL 
REGIONS 

299 

REFERENCES 

302 

BIOGRAPHY 

305 

Chapter  1 


INTRODUCTION 

1 . 1 Introduction  to  Data  Communication  Networks 

A data  communication  network  is  a facility  which  interconnects  a 
number  of  data  devices  (such  as  computers  and  terminals)  by  communica- 
tion channels  for  the  purpose  of  transmission  of  data  between  them. 

Each  device  can  use  the  network  to  access  some  or  all  of  the  resources 
available  throughout  the  network.  These  resources  consist  primarily 
of  computational  power  (CPU  tittlB)  , memory  capacity,  data  bases  and 
specialized  hardware  and  software.  With  the  rapidly  expanding  role 
being  played  by  data  processing  in  today's  society  (from  calculating 
interplanetary  trajectories  to  issuing  electric  bills)  it  is  clear 
that  the  sharing  of  computer  resources  is  a desirability.  In  fact,  the 
distinguished  futurist  Herman  Kahn  of  the  Hudson  Institute  has  forecast 
that  the  "marriage  of  the  telephone  and  the  computer"  will  be  one  of 
the  most  socially  significant  technological  achievements  of  the  next 
two  hundred  years. 

Research  in  areas  related  to  data  communication  networks  began  in 
the  early  1960 's  and  has  blossomed  into  a sizable  effort  in  the  1970's. 
The  ARPANET  (Advanced  Research  Projects  Agency  NETwork)  was  implemented 
in  1970  as  an  experimental  network  linking  a variety  of  university. 
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industrial  and  government  research  centers  in  the  United  States.  Cur- 
rently, the  network  connects  about  one  hundred  computers  throughout 
the  continental  United  States,  Hawaii  and  Europe.  The  network  has 
enjoyed  considerable  success  and  as  a result  several  other  major  net- 
works are  presently  being  planned. 

Following  Kleinrock  [1976] , we  now  describe  the  basic  components 
of  a data  coir  mication  network  and  their  functions.  Fundamentally, 
what  is  known  as  the  communication  subnetwork  consists  of  a collection 
of  nodes  which  exchange  data  with  each  other  through  a set  of  links. 
Each  node  essentially  consists  of  a minicomputer  which  may  have  data 
storage  capability  and  which  serves  the  function  of  directing  data 
which  passes  through  the  node.  The  links  are  data  transmission  chan- 
nels of  a given  data  rate  capacity.  The  data  devices  which  utilize 
the  communication  subnetwork,  known  as  users,  insert  data  into  and 
receive  data  from  the  subnetwork  through  the  nodes.  See  Figure  1.1. 

A more  detailed  description  of  the  network  in  the  context  of  the 
analysis  of  this  thesis  is  presented  in  Chapter  2. 

The  data  travelling  along  <"he  links  of  the  network  is  organized 
into  messages,  which  are  groups  of  bits  which  convey  some  information. 
One  categorization  of  networks  differentiates  those  which  have  message 
storage  at  the  nodes  from  those  which  do  not.  Those  with  storage  are 
known  as  store-and- forward  networks.  Another  classification  is  made 


according  to  the  manner  in  which  the  messages  are  sent  through  the 
network.  In  a circuit-switching  network,  one  or  more  connected  chains 
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of  links  is  set  up  from  the  node  of  origin  to  the  destination  node  of 
of  the  message,  and  certain  proportions  of  data  traffic  between  the 
origin  and  destination  are  then  transmitted  along  these  chains.  The 
other  category  includes  both  message  switching  and  packet  switching 
networks.  In  message-switching,  only  one  link  at  a time  is  used  for 
the  transmission  of  a given  message.  Starting  at  the  source  node  the 
message  is  stored  in  the  node  until  its  time  comes  to  be  transmitted 
on  an  outgoing  link  to  a neighboring  node.  Having  arrived  at  that  node 
it  is  once  again  stored  until  being  transmitted  to  the  next  node.  The 
message  continues  to  traverse  links  and  wait  at  nodes  until  it  finally 
reaches  its  destination.  Packet-switching  is  fundamentally  the  same 
as  message  switching,  except  that  a message  is  decomposed  into  smaller 
pieces  of  maximum  length  called  packets.  These  packets  are  properly 
identified  and  work  their  way  through  the  network  in  the  fashion  of 
message  switching.  Once  all  packets  belonging  to  a given  message  arrive 
at  the  destination  node,  the  message  is  reassembled  and  delivered  to 
the  user.  In  this  thesis  we  shall  be  concerned  with  store-and-forward 
netv;orks  which  employ  message (packet) -switching. 

1 . 2 Discussion  of  the  Message  Routing  Problem 

There  is  a myriad  of  challenging  problems  which  must  be  confronted 
in  the  design  and  operation  of  data  communication  networks.  Just  to 
name  a few,  there  are  the  problems  of  topological  design  (cost  effective 


allocation  of  links  and  their  rate  capacities) , detection  of  equipment 
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failures  and  reconfiguration,  and  the  routing  of  messages  through  the 
network  from  their  nodes  of  origin  to  their  nodes  of  destination.  The 
latter  problem  is  one  of  the  fundamental  issues  involved  in  the  opera- 
tion of  networks  and  as  such  has  received  considerable  attention  in  the 
data-communication  network  literature.  It  is  clear  that  the  efficiency 
with  which  messages  are  sent  to  their  destinations  determines  to  a great 
extent  the  desirability  of  networking  data  devices.  The  subjective  term 
"efficient"  may  be  interpreted  mathematically  in  many  ways,  depending  on 
the  specific  goals  of  the  network  for  which  the  routing  procedure  is 
being  designed.  For  excimple*  ohe  may  wish  to  minimize  total  message 
delay,  maximize  message  throughput,  cost.  etc.  In  general,  this  issue 
is  referred  to  as  the  routing  problem.  We  shall  be  concerned  with  the 
minimum  delay  message  routing  problem  in  this  thesis. 

Routing  procedures  can  be  classified  according  to  how  dynamic 
they  are.  At  one  end  of  the  scale  we  have  purely  static  strategies  in 
which  fractions  of  the  message  traffic  at  a given  node  with  a given 
destination  are  directed  on  each  of  the  outgoing  links,  where  the  frac- 
tions do  not  change  with  time.  On  the  other  end  of  the  scale  we  have 
completely  dynamic  strategies,  which  allow  for  continual  changing  of 
the  routes  as  a function  of  time,  and  also  as  a function  of  message 
congestion  and  traffic  requirements  in  the  network.  Static  procedures 
are  easier  to  implement  tlian  dynamic  ones,  but  lack  the  important 
ability  to  cope  with  changing  congestion  and  traffic  requirements  in 
the  network  possessed  by  dynamic  strategies. 
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Although  a large  variety  of  routing  procedures  have  been  developed 
and  implemented  in  existing  networks  (such  as  AFPANET)  the  lack  of  a 
basic  model  and  theory  able  to  accommodate  various  important  aspects  of 
the  routing  problem  has  made  it  necessary  to  base  the  procedures  almost 
solely  on  intuition  and  heuristics.  Of  concern  here  is  the  fact  that 
previous  techniques  have  been  addressed  primarily  to  static  routing 
strategies,  ’hich  lack  the  previously  discussed  advantages  of  dynamic 
strategies.  The  best  known  approach  to  static  message  routing  in 
store-and- forward  data  communications  networks  is  due  to  KLeinrock 
[1964].  This  approach  is  based  upon  queueing  theory  and  we  describe 
the  principal  elements  here  briefly  for  comparison  with  the  dynamic 
method  which  we  shall  discuss  subsequently: 

(a)  The  messages  arrive  from  users  to  the  network  according  to 
independent  constant  rate  Poisson  processes  and  their  lengths  are 
assumed  to  be  independent  exponentially  distributed  and  independent  of 
their  arrival  times. 

(b)  At  subsequent  nodes  along  the  paths  of  the  messages,  the 
lengths  of  the  messages  and  their  interarrival  times  become  dependent, 
a fact  which  makes  the  analysis  extremely  difficult.  To  cope  with  this 
problem,  the  famous  independence  assumption  is  introduced,  requiring  the 
messages  to  "lose  their  identities"  at  each  node  and  to  be  assigned 

new  independent  lengths. 

(c)  Once  a message  arrives  at  a node,  it  is  first  assigned  to  one 


of  the  outgoing  links  and  waits  in  queue  at  that  link  until  it  is 


r 
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transmitted. 

(d)  Based  on  queueing  analysis,  the  average  delay  in  steady  state 
experienced  by  messages  in  each  link  is  calculated  explicitly  in  terms 
of  the  various  arrival  rates,  average  message  length  and  the  capacity 
of  the  link. 

(e)  A routing  procedure  is  then  found  to  minimize  the  average 
delay  ove..  the  entire  network.  See  Cantor  and  Gerla  [1974]  . 

The  routing  procedure  thus  obtained  is  static , namely  constant  in 
time  cind  a function  only  of  the  various  average  parameters  of  the  sys- 
tem. In  the  parlance  of  control  theory,  which  is  the  language  which  we 
shall  be  using  for  much  of  this  thesis,  such  a strategy  is  referred  to 
as  open- loop. 

Since  Kleinrock's  model  was  proposed  in  1962,  researchers  in  the 

4 

t 

area  have  repeatedly  expressed  the  desire  to  find  approaches  to  other 
aspects  of  the  routing  problem.  In  particular,  the  goal  is  to  find 
an  approach 

(i)  in  which  the  independence  assumption  is  not  required,  ly 

(ii)  that  will  be  able  to  handle  transients  and  dynamical  situa- 
tions and  not  only  steady  state,  and 

(iii)  that  can  lead  to  optimal  closed-loop  control  strategies, 
namely  strategies  that  change  according  to  the  congestion 
in  the  network. 

Requirement  (i)  is  desirable  since  the  independence  assumption  may 
be  quite  inappropriate  in  various  situations.  Furthermore,  it  is  not 
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easy  to  assess  the  validity  of  this  assumption  for  a given  network. 

The  desirability  of  requirement  (ii)  has  been  discussed  previously. 
Finally,  perhaps  the  most  important  requiremeni.  is  (iii),  for  it  is  a 
fundamental  fact  of  optimal  control  theory  that  closed-loop  strategies 
are  much  less  sensitive  than  open-loop  ones  to  perturbations  in  the 
parameters  of  the  system.  Hence,  occurrences  such  as  link  and  node 
failures  or  u ■ expected  bursts  of  user  input  to  the  network  are  accom- 
modated much  better  by  closed-loop  strategies. 

It  is  pointed  out  in  Segall  [1976]  that  tlie  traditional  queueing 

theory  approach  can  in  principle  be  adapted  to  closed  loop  strategies, 

but  that  the  number  of  states  required  would  be  immense.  This  approach 

looks  at  each  message  or  packet  as  an  entity,  and  therefore  the  state 

of  the  network  is  described  by  the  number  and  destination  of  messages 

(packets)  in  eacli  of  the  buffers.  Therefore,  in  a network  with  N nodes, 

[i  outgoing  links  per  node  in  the  average  and  buffers  of  maximum  capacity 

N-1 

of  M messages  (packets),  the  number  of  states  is  approximately  (pNM)  , 
which  is  an  extremely  large  number  even  for  the  smallest  of  networks. 

1 Segall  [1976]  lias  introduced  a model  for  message  routing  whicli 

j is  capable  of  achieving  requirements  (i)-(iii)  above.  His  approach  is 

to  model  the  flow  of  messages  in  a message  (packet) -switched  store-and- 
forward  data  communication  network  in  a continuous  dynamical  state 
space  setting.  The  continuous  nature  of  the  state  is  justified  by 
recognizing  tliat  any  individual  message  contributes  very  little  to  the 


overall  beliavior  of  ttie  network,  so  that  it  is  not  necessary  to  look 


individually  at  each  of  the  messages  and  their  lengths.  In  this  vein, 
it  makes  more  sense  to  regard  the  network  in  the  more  macroscopic 
fashion  of  Segall's  model. 

Having  established  the  model,  Segal 1 expresses  the  minimum  delay 
dynamic  routing  problem  as  a linear  optimal  control  problem  with  linear 
state  and  control  variable  inequality  constraints  for  which  a feedback 
solution  i ■ sought.  Should  such  a solution  to  this  problem  be  obtained, 
the  resulting  strategy  would  be  dynamic  and  closed-loop.  The  model  and 
associated  optimal  control  problem  are  discussed  briefly  in  the  next 
section  and  are  presented  in  detail  in  Chapter  2. 

1. 3 Thesis  Overview 
1,3.1  Objective  of  Thesis 

The  goal  of  this  research  is  to  obtain  feedback  solutions  to  the 
linear  optimal  control  problem  with  linear  state  and  control  variable 
inequality  constraints  suggested  by  Segall  for  the  message  routing 
problem.  Undoubtedly,  the  principal  difficulty  presented  by  this  prob- 
lem is  the  presence  of  state  variable  inequality  constraints.  This 
contention  is  supported  by  the  observation  that  very  few  feedback  solu- 
tions have  been  found  for  optimal  control  problems  with  this  type  of 
constraint.  We  clearly  shall  have  to  exploit  the  special  properties 
of  our  problem  (such  as  linearity  and  the  structure  of  the  dynamics  and 
constraints)  to  develop  new  theory  and  techniques  capable  of  producing 


feedback  solutions. 


1.3.2  Approach 


We  begin  our  discussion  of  approach  by  briefly  describing  Segall's 
state  space  model.  In  the  model  the  state  variables  represent  the  quan- 
tity of  messages  stored  at  the  nodes,  distinguished  according  to  node 
of  ciirrent  residence  and  node  of  ultimate  destination.  In  reality,  the 
measure  of  message  storage  at  the  nodes  is  a discrete  variable  (such  as 
number  of  me:  ages,  packets,  bits,  etc.).  However,  in  the  spirit  of 
viewing  the  network  from  a macroscopic  point  of  view,  we  assume  that 
the  units  of  data  traffic  are  such  that  after  appropriate  normalization 
the  states  may  be  approximated  closely  by  contin>ious  variables.  The 
control  variables  represent  the  flow  rate  of  traffic  in  the  links,  where 
each  control  represents  that  portion  of  a given  link's  rate  capacity 
devoted  to  transmitting  messages  of  a given  destination.  Finally,  the 
inputs  are  the  flow  rates  of  messages  entering  the  network  from  the 
users.  In  this  thesis,  we  consider  the  inputs  to  be  deterministic 
functions  of  time.  The  dynamical  equation  which  represents  the  flow  of 
messages  in  the  network  is 

x(t)  = B u(t)  + ait)  (1.1) 

where  x(t) , u(t)  and  a^(t)  are  the  vectors  of  state  variables,  control 
variables  and  inputs  respectively.  In  the  static  flow  literature  the 
matrix  B is  known  as  the  "incidence  matrix",  and  it  is  composed  solely 
of  +l's,  -I's  and  O's  which  capture  the  node-link  configuration  of  the 
network.  We  also  have  several  essential  constraints.  The  state  varia- 


-19- 


ble  inequality  constraints  are  x(t)  ^ 0^  for  all  t,  since  the  message 
storage  must  always  be  non-negative.  The  control  variable  inequality 
constraints  are  uit)  t U for  all  t,  where  U = {u:u  ^ £ and  £ u S • 
Here  D is  a matrix  of  O's  and  +l's  and  C is  a vector  of  link  rate 
capacities.  These  constraints  represent  flow  non-negativity  and  link 
rate  capacity  constraints  respectively. 

We  row  associate  with  the  dynamical  state  space  model  the  linear 
cost  functional 


J 


a x(t)dt 


(1.2) 


where  3t(tQ)  = 3^  is  given,  t^  is  given  implicitly  by  x(t^)  = £ and  a 
is  a column  vector  of  constant  weighting  factors.  The  implications  of 
the  stipulation  that  3c(t^)  = 0 are  discussed  in  Section  2.4.  We  note 
for  now  that  whv.r.  a is  all  I's,  then  J is  exactly  the  total  delay  ex- 
perienced by  all  of  the  messages  traveling  through  the  network  on 
[tj^,  tf]-  By  adjusting  the  values  of  the  elements  of  a,  J may  be  made 
to  represent  a desired  form  of  weighted  total  delay.  The  optimal 
control  problem  which  represents  the  dynamic  feedback  message  routing 
problem  is: 

Find  the  control  u(t)  as  a function  of  time  and  state, 
u(t)  E ^(t,  x)  , that  will  bring  the  state  from 
(given)  to  x(t^)  = £ while  minimizing  J subject  to  the  dynair.- 
ics  and  state  and  control  variable  inequality  constraints. 
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Our  approach  to  solving  the  above  problem  shall  now  be  described 
briefly.  We  begin  by  deriving  the  necessary  conditions  of  optimality 
associated  with  the  optimal  control  problem  and  prove  that  they  are 
also  sufficient.  Realizing  that  there  is  extremely  little  hope  of  ob- 
taining at  this  time  a feedback  solution  for  the  general  deterministic 
input  problem,  we  restrict  the  inputs  to  be  constant  in  time.  The 
necessary  corditions  indicate  that  with  this  assumption  the  optimal 
feedback  control  is  regior.wise  constant  over  the  state  space.  We  then 
develop  a procedure  which  utilizes  the  necessary  conditions  to  construct 
all  of  these  regions  and  identify  their  associated  optimal  controls. 
Although  the  procedure  is  not  readily  implementable  on  the  conputer  for 
problems  involving  general  multi-destination  networks,  we  are  able  to 
utilize  special  structural  properties  of  the  problem  to  devise  a com- 
puter algorithm  for  problems  involving  single  destination  networks  with 
all  unity  weightings  in  the  cost  functional. 

1.3.3  Preview  of  Results 

In  this  section  we  elaborate  on  the  discussion  of  approach  of  the 
previous  section  and  simultaneously  describe  the  highlights  of  the 
results  which  are  obtained. 

According  to  the  necessary  conditions  which  are  derived  for  the 
problem,  any  optimal  control  must  satisfy 

U*(t)  = ARG  MIN  (V^(t)B  u(t))  (1.3) 

u(t)etl 

pointwise  in  time.  Here,  ^(t)  is  the  vector  of  costate  variables  cor- 
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responding  to  the  state  x(t)  and  is  given  by  a linear  differential 
equation  with  free  terminal  conditions.  Of  particular  interest  is  the 
fact  that  a costate  may  exhibit  discontinuities  at  times  when  its 
associated  state  is  traveling  on  a boundary  arc  x = 0.  We  are  also 
able  to  prove  the  significant  result  that  the  necessary  conditions  of 
optimality  are  also  sufficient. 

For  given  values  of  the  costate  vector,  the  pointwise  minimization 
is  a linear  program.  Owing  to  fundamental  properties  of  linear  pro- 
gramming we  are  able  to  deduce  the  following:  (i)  the  optimal  control 
always  lies  at  boundary  points  of  U,  and  therefore  is  of  the  bang-bang 
variety;  (ii)  non-uniqueness  of  the  optimal  control  is  a possibility; 
(iii)  the  minimization  need  not  be  performed  at  every  time,  but  may  be 
solved  parametrically  in  time  to  find  the  optimal  bang-bang  controls  and 
switch  times. 

Note  that  all  of  the  above  results  apply  for  general  deterministic 
inputs.  Henceforth  we  restrict  ourselves  to  the  special  case  in  which 
the  inputs  are  constant  in  time.  In  this  case,  the  bang-bang  nature  of 
the  optimal  control  implies  that  the  slope  of  the  optimal  state  trajec- 
tory is  piecewise  constant  in  time.  We  are  now  able  to  state  and  prove 
sufficiency  conditions  under  which  any  initial  state  x(tQ)  = is 
controllable  to  x(t^)  = £ for  a given  network  with  given  set  of  constant 
inputs. 

Using  the  necessary  conditions,  we  are  able  to  show  tluit  the 
optimal  control  is  regionwise  constant  and  that  the  regions  are  convex 


polyhedral  cones  in  the  state  space.  A procedure  is  then  developed 


-22- 


which  employs  the  spirit  of  dynamic  programming  to  construct  these  cones 
and  their  associated  optimal  controls  by  a sequence  of  backward  optimal 
trajectories  propagating  from  the  final  time  t^.  When  a sufficiently 
large  variety  of  these  trajectories  is  utilized  to  construct  regions, 
the  union  of  these  regions  fills  up  the  state  space  with  optimal  con- 
trols, thus  constituting  the  feedback  solution. 

In  order  to  solve  for  the  backward  optimal  trajectory,  v/e  must 
propagate  the  costates  backward  in  time  according  to  their  differential 
equation  and  solve  the  minimization  problem  (1.3)  for  all  solutions. 
However,  there  is  a question  regarding  the  appropriate  values  of  the 
costates  which  realize  a particular  backward  optimal  trajectory  through 
(1.1)  and  (1.3).  The  resolution  to  this  problem  is  geometrical  in 
nature  in  that  it  considers  the  Hamiltonian  function  associated  with 
the  necessary  conditions  to  be  a hyperplane  which  is  continuously  rota- 
ting about  the  constrained  region  of  message  flow  while  remaining 
tangent  to  it.  As  the  costates  are  the  coefficients  of  the  hyperplane, 
their  appropriate  values  are  determined  by  orienting  the  hyperplane  in 
a prescribed  fashion  with  respect  to  the  constraint  region.  This  argu- 
ment freely  employs  geometrical  concepts  in  linear  programming. 

We  are  unable  to  devise  a conputer  algoritlim  to  implement  the 
above  procedure  for  general  multi-destination  network  problems  due  to 
several  complicating  properties.  However,  we  are  able  to  show  that  for 
problems  involving  single  destination  networks  with  all  unity  weightings 


in  the  cost  functional  these  complicating  properties  do  not  apply.  A 
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computational  implementation  of  the  procedure  is  then  readily  formu- 
lated and  a con^juter  example  is  performed.  It  turns  out  that  one  of 
the  computational  tasks  of  the  algorithm  is  extremely  inefficient. 

This  task  is  involved  with  the  problem  of  finding  all  of  the  non-unique 
extremal  solutions  to  a linear  program.  As  a consequence,  the  computa- 
tional feasibility  of  the  algorithm  is  contingent  upon  the  development 
of  a more  efficient  technique  for  solving  this  problem. 

In  evaluating  the  ultimate  desirability  of  the  feedback 
approach,  we  must  take  into  account  the  fact  that  considerable  computer 
storage  may  be  required  to  implement  the  feedback  solution  on  line. 

This  results  from  the  necessity  of  storing  all  of  the  linear  inequali- 
ties which  specify  the  convex  polyhedral  cones  in  the  state  space, 
and  there  may  be  many  such  cones.  A tradeoff  is  therefore  in  order 
between  the  storage  involved  to  implement  closed-loop  solutions  and 
the  computation  involved  in  the  implementation  of  open  loop  solutions. 

1.4  Synopsis  of  Thesis 

The  purpose  of  this  section  is  to  provide  a brief  summary  of 
the  remaining  chapters. 

Chapter  2 

The  state  space  model  and  optimal  control  problem  of  Segall  are 
described  in  detail.  A brief  discussion  is  devoted  to  two  possible 
open-loop  solutions:  The  first  employs  linear  programming  through  dis- 
cretization in  time  and  the  second  is  an  adjoined  penalty  function 
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technique.  Note  that  although  they  are  open- loop,  these  techniques 
are  dynamic. 

Chapter  3 

In  this  chapter  we  present  the  procedure  for  the  synthesis  of  a 
feedback  solution  to  the  optimal  control  problem  with  constant  inputs. 

We  begin  by  deriving  the  necessary  conditions  of  optimality  and  prove 
that  they  are  sufficient.  Basea  upon  these  conditions  we  characterize 
the  feedback  regions  of  interest  as  convex  polyhedral  cones.  After 
providing  a few  motivating  examples , we  present  the  algorithm  for  the 
backward  construction  of  the  feedback  control.  Finally,  we  isolate 
and  describe  in  detail  those  properties  of  the  algorithm  which  compli- 
cate its  computational  implementation. 

Chapter  4 

The  geometrical  linear  programming  interpretation  is  utilized  to 
construct  proofs  that  the  complicating  properties  of  the  algorithm  of 
Chapter  3 do  not  apply  for  problems  involving  single  destination  net- 
works with  all  unity  weightings  in  the  cost  functional.  This  fortuitous 
situation  enables  us  to  construct  an  algorithm  which  is  implementable 
on  the  computer.  An  example  is  run  for  a five  node  network.  We  then 
present  thoughts  on  how  the  techniques  of  this  chapter  may  be  applied 
to  general  multi-destination  network  problems  with  non-unity  weightings 


in  the  cost  functional. 
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Chapter  5 

In  this  chapter  we  comment  on  various  aspects  of  the  approach 
in  general  and  of  the  feedback  control  algorithm  in  particular.  Some 
insight  is  provided  into  related  topics  not  specifically  discussed  in 
preceding  chapters.  We  then  summarize  our  results,  present  the  spe- 
cific contributions  and  make  suggestions  for  further  work  in  this 
area. 


Chapter  2 

NEW  DATA  COMMUNICATION  NETWORK  MODEL  AND  OPTIMAL 
CONTROL  PROBLEM  FOR  DYNAMIC  ROUTING 

2. 1 Introduction 

In  this  chapter,  the  flow  and  storage  of  messages  in  a message- 
(packet) -switched  store-and- forward  data  communication  network  are  ex- 
pressed in  a dynamical  state  space  setting.  This  model  for  routing  was 
introduced  by  scgall  [1976].  The  principal  elements  of  the  model  - state 
variables,  control  variables,  and  inputs  - are  defined  to  represent 
mathematiCw^iiy  the  fundamentals  of  network  operation:  storage,  traffic 
flow  and  message  input  respectively.  Emerging  from  this  characteriza- 
tion is  an  ordinary  linear  vector  differential  equation  which  dynami- 
cally describes  the  storage  state  of  the  network  at  every  time.  State 
varicUale  positivity  constraints  and  control  variable  capacity  con- 
straints, both  linear,  are  imposed  as  an  essential  part  of  the  model. 
Presently,  we  assume  that  the  inputs  are  deterministic  functions  of 
time,  representing  a scheduled  rate  of  demand. 

Arising  naturally  out  of  the  model  as  defined  is  a linear  inte- 
gral cost  functional  which  is  equal  to  the  total  delay  experienced  by 
H all  of  the  messages  travelling  in  the  network.  Owing  to  the  generality 

B of  our  expression,  we  are  also  able  to  formulate  a linear  cost  function- 
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work.  Combining  tile  minimization  of  the  cost  functional  with  the  dy- 
namics and  associated  constraints,  we  obtain  a linear  optimal  control 
problem  with  linear  state  and  control  variable  inequality  constraints  for 
which  we  seek  a feedback  solution.  We  then  discuss  in  some  detail  how 
the  assumptions  inherent  in  the  optimal  control  problem  formulation  re- 
late to  the  real  world  situation  of  data  communication  network  message 
routing  op<  '-ation. 

The  advantages  of  this  approach  were  discussed  in  Section  1.2. 

The  principal  disadvantage  is  cis  :!  i.iated  with  th^’  difficulty  of  solving 
state  variable  inequality  constrained  optimal  control  problems.  This 
is  particularly  true  when  a feedback  solution  is  sought.  The  problem 
is  placed  rn  perspective  by  providing  a summary  of  the  previous  work 
which  has  been  performed  in  this  area  and  pointing  out  the  need  for  the 
development  of  additional  theory  for  our  particular  formulation.  This 
discussion  sets  the  stage  for  the  contributions  to  this  problem  area 
which  are  achieved  in  subsequent  chapters. 

Although  the  principal  goal  of  the  thesis  is  to  obtain  closed- 
loop  solutions  to  the  optimal  control  problem,  there  are  several  ap- 
proaches to  obtaining  open-loop  solutions  which  are  conceptually 
straightforward  and  are  therefore  of  some  interest.  Two  such  approaches 
are  reported  on  briefly  at  the  end  of  this  chapter:  linear  programming 
through  discretization  and  a penalty  function  method. 
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2.2  Basic  Elements  of  the  State  Space  Model 
2.2.1  Topological  Representation 

We  visualize  data  networks  graphically  to  consist  of  a collection 
of  nodes  connected  by  a set  of  links  between  various  pairs  of  the  nodes. 
In  Section  1.1  we  presented  a discussion  of  the  general  function  of 
nodes  and  links  in  the  network.  In  our  model,  we  shall  assume  that  the 
links  are  simplex,  that  is,  carry  messages  in  one  direction  only  from 
the  node  at  the  input  end  to  the  node  at  the  output  end. 

In  a network  consisting  of  N nodes  we  associate  with  each  node 
an  integer  in  the  set  {l,  2,  ...,  n}  and  denote  this  collection  of  nodes 
by  W.  The  link  connecting  node  i to  node  k is  denoted  by  (i,k),  and 
the  collection  of  all  links  in  the  network  is 

Notation  2.1  L = {(i,k) , such  that  i,k  e W and  there  is  a direct 
link  connecting  i to  k}. 

We  now  denote 

A 

Notation  2.2  = capacity  of  link  (i,k)  in  units  of  traffic/unit 

time,  (i,k)  EL 

and  for  every  i C SI  denote 

Notation  2.3  E(i)  = collection  of  nodes  k such  that  (i,k)  e L, 

I(i)  = collection  of  nodes  S!.  such  that  (il,i)  E L. 

In  Figure  2.1  we  depict  a graphical  representation  of  such  a data  com- 
munication network. 


Fiyure  2.1  Data  Communicatioii  Network  Topology 

2.2.2  Message  Flow  and  Storage  Representation 

In  accordemce  with  the  operational  philosophy  of  the  network, 
the  users  at  each  node  in  the  network  may  input  messages  whose  ultimate 
destinations  are  any  of  the  other  nodes  in  the  network.  We  characterize 
this  message  traffic  flow  input  to  the  network  by: 

Notation  2.4  a|(t)  = rate  of  traffic  with  destination  j arriving 


F 
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at  node  i (from  associated  users)  at  time  t. 

The  message  traffic  from  each  user  is  first  fed  into  the  associated  node 
and  either  immediately  transmitted  on  an  outgoing  link  or  stored  for 
eventual  transmission.  Also,  each  node  may  serve  as  an  intermediate 
storage  area  for  messages  entering  on  incoming  links  enroute  to  their 
destinations.  Once  a message  reaches  its  destination  node,  it  is  im- 
mediately forwarded  to  the  appropriate  user  without  further  storage. 

Hence,  at  each  node  i e W of  the  network  at  any  point  in  time  we 
may  have  messages  in  residence  whose  destinations  are  all  nodes  other 
than  i.  Let  us  now  imagine  that  at  each  node  i c SI  we  have  N-1  "boxes"; 
and  that  in  each  of  these  boxes  we  place  all  the  traffic  (messages, 
packets,  bits,  etc.)  whose  destination  is  a particular  node,  regardless 
of  its  origin.  We  do  this  for  all  possible  destinations  1,  2,  ...  (i-1)  , 
(i+1) , ...N.  We  now  define  the  state  variables  of  our  model  as 

Notation  2.5  x^ (t)  = amount  of  traffic  at  node  i at  time  t whose 
1 

final  destination  is  node  j , where 
i,j  e W,  i / j. 

The  amount  of  traffic  residing  in  each  box  at  any  time  t is  mea- 
sured in  some  arbitrary  unit  (messages,  packets,  bits,  etc.).  Strictly 
speaking,  the  states  are  therefore  discrete  variables  with  quantization 
level  determined  by  the  particular  unit  of  traffic  selected.  However, 
we  shall  assume  that  the  units  are  such  that  after  appropriate  normali- 
zation the  states  x^  can  be  approximated  by  continuous  variables.  The 


4 


rationale  underlying  this  approximation  is  presented  in  Section  1.2. 
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We  simply  repeat  here  that  this  macroscopic  point  of  view  regarding 
messages  is  justifiable  in  relation  to  the  overall  goals  of  desiraible 
network  operation.  Note  that  in  a network  consisting  of  N nodes,  the 
maximum  number  of  state  variables  as  defined  above  is  N(N-l),  which  most 
certainly  is  reasonable  when  compared  to  the  huge  number  of  states 
associated  with  finite-state  models  (see  Section  1.2). 

Then  is  a fundamental  difference  between  the  states  described 

1 

here  and  the  message  queues  of  the  traditional  approach.  In  our  model, 
when  a message  with  destination  j arrives  at  node  i from  either  outside 
the  network  (i.e.,  from  users)  or  from  some  adjacent  node,  it  is  classi- 
fied as  belonging  in  the  "box"  , and  as  such  is  associated  with  the 
node  i.  When  its  time  comes  to  be  sent,  the  routing  strategy  to  be 
developed  in  subsequent  chapters  assigns  the  message  to  be  sent  along 
some  outgoing  link  (i,k),k  e E(i).  In  previous  models  (e.g..  Cantor  and 
and  Gerla  [1974])  messages  arriving  at  node  i are  immediately  assigned 
to  some  outgoing  link  (i,k),k  E E(i),  by  the  routing  strategy,  there  to 
await  transmission.  Hence,  at  least  in  an  intuitive  sense,  the  decision 
as  to  what  direction  to  send  a message  is  made  in  our  model  at  a later 
time,  thus  enabling  the  strategy  in  force  to  make  a more  up-to-date 
decision.  The  ultimate  performance  of  our  strategy  should  benefit  from 
this  characteristic. 

The  final  element  of  our  message  flow  representation  is  the  de- 
scription of  the  allocation  of  link  flow  capacity.  Each  link  emanating 
from  a particular  node  i is  shared  by  some  or  all  of  the  up  to  (N-1) 


-32- 
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types  of  messages  stored  at  time  t at  node  i.  This  now  gives  rise  to 
the  definition  of  the  control  variables  of  our  state  space  model: 

Notation  2.6:  ^ portion  of  the  rate  capacity  of  the  link  (i,k) 

used  at  time  t for  traffic  with  final  destina- 
tion j . 

So  defined,  the  controls  are  the  decision  elements  of  the  model  availcible 
to  be  adjusted  at  the  command  of  the  routing  strategy.  The  designation 
of  the  states  corresponding  to  a particular  node  and  controls  correspond- 
ing to  a particular  link  are  illustrated  in  Figure  2.2. 


Figure  2.2 


Illustration  of  State  and  Control  Variables 
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2. 3 Dynamic  Equations  and  Constraints 

We  are  now  ready  to  state  the  dynamic  relationship  between  the 
elements  of  our  state  space  model.  Assuming  the  inputs  are  determin- 
istic functions  of  time,  the  time  rate  of  change  of  tlie  number  of  mes- 
sages contained  in  each  box  is  given  by 

x|(t)  = a^(t)  - 23  + 23  u^(t)  (2.1) 

^ kEE(i)  ^ Hel(i) 

i,j  L W,  j ^ i. 

That  is,  the  box  corresponding  to  the  state  x^  is  increased  by  the  rate 
of  messages  arriving  from  users  (a^(t))  and  messages  arriving  on  in- 
coming links  (uj^(t) , H e I(i) , ^ ^ j)  and  depleted  by  the  rate  of 
messages  departing  on  outgoing  links  k c Eli)).  A pictorial 

depiction  of  the  message  flow  situation  corresponding  to  equation  (2.1) 
is  given  in  Figure  2.3. 

The  ordinary  differential  equation  formulation  of  equation  (2.1) 
is  valid  for  deterministic  inputs.  When  the  inputs  are  stochastic,  we 
must  express  the  relationship  in  the  incremental  fashion  of  stochastic 
differential  equations  and  properly  define  the  nature  of  the  input  pro- 
cess. However,  the  important  point  here  is  that  the  state  space  de- 
scription of  message  flow  and  storage  presented  in  Section  2.2  is  suffi- 
ciently general  to  accommodate  a wide  variety  of  input  processes. 

An  essential  feature  of  the  model  is  the  set  of  constraints  we 


must  impose  on  the  state  and  control  variables.  In  order  for  tlie 
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mathematical  statement  of  our  problem  to  make  physical  sense,  we  must 
insist  on  non-negativity  of  the  message  storage  state  variables  and  of 
the  flow  control  variables: 


x^(t)  > 0 


V-  t 


(2.2) 


and 


u:',  (t)  > 0 
ik  — 


V-  t. 


(2.3) 


The  rate  capacity  constraints  on  each  transmission  link  is  expressed  by 


E 

jeN 


u^^(t) 

ik 


‘^  •1, 
— ik 


V-  t 

V(i,k)  CL,  j i . 


(2.4) 


Constraints  (2.2)-(2.4)  are  the  only  ones  which  shall  be  dealt  with 
explicitly  in  this  thesis.  The  assumption  is  therefore  made  that  the 
storage  areas  containing  the  messages  corresponding  to  the  state  vari- 
ables are  infinite  in  capacity.  In  practice,  of  course,  these  areas 
v;ill  be  limited  in  size,  so  that  we  may  wish  to  insist  on  upper  bounding 
state  variable  constraints.  Possible  forms  of  these  constraints  are 
x^  ^ or  x^  £ depending  on  the  actual  assignment  of  message 

storage  in  a node. 


2. 4 Performance  Index  and  Optimal  Control  Problem  Statement 

As  indicated  in  Section  1.2,  one  of  the  major  problems  associated 
with  routing  strategies  predicated  upon  queueing  models  is  the  require- 
ment of  the  independence  assumption  in  order  to  derive  the  closed-form 
expression  for  the  total  message  delay  in  the  network.  This  assumption 
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may  be  at  great  variance  with  the  physical  realities  of  the  situation. 

On  the  other  hand,  optimal  control  oriented  formulations  such  as  ours 
require  only  a functional  expression  for  the  quantity  of  interest  in 
terms  of  the  state  and  control  variables  of  the  model.  By  design,  the 
method  in  which  we  have  defined  the  state  variables  gives  rise  to  possi- 
ble performance  indices  which  are  most  appropriate  in  this  application. 
For  example,  serve  that  if  x(t)  is  the  amount  of  message  traffic 
residing  in  some  box  at  time  t,  then  the  quantity 

f 

x(t)dt  (2.5) 


gives  the  total  time  spent  in  this  box  by  the  traffic  passing  through 
it  during  the  time  period  of  interest  [tQ,t^l,  when  t^  is  such  that 
x(t^)  = 0.  Consequently,  expression  (2.5)  is  exactly  the  total  delay 
in  the  box  experienced  by  the  class  of  messages  represented  by  x(t). 
Hence,  the  total  delay  experienced  by  all  the  messages  as  they  travel 
through  the  networ)c  during  t^]  is  given  by 


D 


dt 


(2.6) 


i,j  e N,  j i 


where  t^  is  defined  as  the  time  at  which  all  the  message  storage  state 

variables  x?  go  to  zero.  Priorities  Ccin  be  accommodated  in  the  cost 
1 

functional  (2.6)  by  associating  non-equal  weightings  to  the  appro- 
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priate  state  varieibles,  so  that  we  have 


pf 

" J 

E aW"' 

(2.7) 


j e N / j i 


with  t^  defined  as  above.  A logical  fashion  in  which  to  assign  priori- 
ties is  by  destination,  in  which  case  V-  k,)l  ^ j.  Cost  func- 

tional (2,7)  is  then  a measure  of  total  delay  weighted  by  destination. 
We  now  have  all  the  elements  needed  to  state  our  optimal  control  prob- 
lem. In  words,  the  data  communication  network  dynamic  message  routing 
problem  is: 


At  every  time  t,  given  the  knowledge  of  the  traffic 
congestion  in  the  network  (x?(t),  i,j  e N,  j i)  , dynami- 
cally decide  what  portion  of  each  link  capacity  to  use  for 
each  type  of  traffic  (i.e.,  assign  u^  (t) , (i,k)  e L,  j c W)  , 

XK 

so  as  to  minimize  the  specified  cost  functional  (i.e.,  total 
delay  if  a|  = 1 V-  i,j,  i ^ j)  while  bringing  the  traffic 
from  a specified  initial  level  to  zero  at  the  final  time. 

To  facilitate  the  expression  of  this  problem  in  compact  matliemati- 
cal  form,  we  define  the  five  column  vectors  - £,  )c,  \j,  C and  a - whicn 
are  respectively  concatenations  of  the  inputs,  state  variables,  control 
variables,  link  capacities  and  cost  functional  weightings.  Denote 
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] 

r,-  _ 


I 


'Jk.  i 


n = dimension  (a)  = dimension  (jc)  = dimension  (a)  , m = dimension  (ii)  and 

r = dimension  (£)  = card  (L) . For  a given  network  topology  we  define 

the  n ^ m matrix  B as  follows:  associated  with  every  state  variable 

T 

IS  a row  b of  B such  that 


b u 


-E 

k£E(i) 


ik 


^ E -I 


j e W,  j ^ i. 


(2.8) 


The  matrix  B is  analogous  to  the  incidence  matrix  which  describes 

flow  in  a static  network.  However,  a fundamental  distinction  from  the 

static  flow  situation  is  that  we  do  not  require  conservation  of  flow  at 

nodes  as  we  have  the  capability  of  message  storage  in  nodes.  Note  that 

B is  composed  entirely  of  +l's,  -I's  and  O's  and  that  every  column  of 

B has  at  most  two  non-zero  elements.  If  a particular  column  has  exactly 

one  non-zero  entry  then  it  is  -1. 

Similarly,  we  define  the  r x m matrix  D:  associated  with  every 

T 

link  (i,k)  is  a row  d of  II  such  that 

1 


& 


y 


represents  the  constraint  (2.4)  . The  elements  of  D are  O's  and  +l's 
only,  and  each  column  has  precisely  one  +1. 

We  may  now  compactly  express  the  linear  optimal  control  problem 
with  linear  state  and  control  variable  inequality  constraints  whicli 
represents  the  data  communication  network  dynamic  message  routing 


4 
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problem  previously  stated: 


Find  the  set  of  controls  u as  a function  of  time  and 


state 


u(t)  = u(t,  x)  t e [t^,  t^] 


(2.10) 


that  will  bring  the  state  from  a given  initial  condition 


x(tQ/  = Xq  to  “ 9.  minimizes  the  cost  functional 


I 


(t)]dt 


(2.11) 


subject  to  the  state  dynamics 


x(t)  = B u(t)  + a(t) 


(2.12) 


and  constraints  on  the  state  and  control  variables 


x(t)  >0  V t e [tp,  t^l 


(2.13) 


D u < C 


u > 0. 


(2.14) 


Note  that  a(t)  must  be  such  that  the  state  is  controllable  to  zero 
with  the  availcdjle  controls.  Conditions  under  whic)i  this  is  true  are 
given  for  a special  case  in  Section  3.2.2. 

2 . 5 Discussion  of  the  Optimal  Control  Problem 

Before  engaging  in  the  details  of  tlie  solution  to  ( 2. 10)  - (2 . 14)  , 
some  discussion  regarding  tlie  validity  of  the  problem  statement  is 
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appropriate.  To  begin,  we  have  stipulated  inputs  which  are  known  expli- 
citly as  a function  of  time,  whereas  computers  certainly  operate  in  a 
stochastic  user  demand  environment.  Secondly,  the  requirement  that  all 
the  storage  states  go  to  zero  at  some  final  time  is  not  consistent  with 
a network  continually  receiving  input  and  storing  messages  in  a steady 
fashion.  Also,  as  pointed  out  in  Section  2.3,  we  have  ignored  upper 
bounds  on  mes-  age  storage  capacity.  Finally,  by  specifying  a feedback 
function  of  the  type  (2.10)  we  are  assuming  that  total  information  re- 
garding storage  is  available  throughout  the  entire  network  for  controller 
decisions.  In  practice,  one  may  wish  to  consider  schemes  which  allow 
for  the  control  decision  to  be  made  on  the  basis  of  local  information 
only,  so  called  distributed  control  schemes. 

With  the  above  drawbacks  in  mind,  we  now  provide  justification  for 
our  approach.  We  begin  by  pointing  out  that  none  of  the  assumptions 
made  thus  far  are  inherent  in  a basic  state  space  model.  These  have 
simply  been  invoked  to  provide  a problem  formulation  for  which  there 
exists  some  hope  of  obtaining  a reasonable  solution  at  this  early  stage 
of  experience  with  the  model.  Also,  they  may  not  be  as  limiting  as 
they  first  appear.  For  instance,  one  possible  approximation  for  the 
situation  with  stochastic  inputs  is  to  take  into  account  only  the  en- 
semble average  rates  of  the  inputs.  We  then  design  the  routing  strategy 
by  solving  (2. 10) - (2. 14)  with  these  averages  serving  as  the  determinis- 
tic inputs  £{t) , and  employ  the  controls  thus  obtained  in  the  operation 
of  t)ie  network.  Such  a strategy  may  prove  to  be  reasonably  successful 
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if  the  variances  and  higher  moments  of  the  distributions  of  the  inputs 
are  small  compared  to  the  means. 

Next,  the  requirement  that  all  states  go  to  zero  at  the  final  time 
may  correspond  to  a situation  in  which  one  wishes  to  dispose  of  message 
backlogs  at  the  nodes  for  the  purpose  of  temporarily  relieving  conges- 
tion in  the  network  locally  in  time.  This  procedure  may  represent  that 
portion  of  in  overall  scheme  during  which  inputs  are  appropriately  regu- 
lated or  no  longer  forthcoming.  In  the  latter  case  we  may  refer  to  the 
resulting  operation  as  a "minimum  delay  close  down  procedure". 

Elimination  of  the  state  variable  upper  bounds  is  not  always 
limiting,  as  we  shall  discover  for  a class  of  single  destination  network 
problems  studied  in  Chapter  4.  In  this  case,  the  optimal  routing  strat- 
egy never  requires  any  state  to  exceed  its  initially  specified  value, 
which  certainly  must  be  within  the  available  storage  capacity. 

Finally,  the  assumption  of  a centralized  controller  may  well  be 
valid  in  the  case  of  a small  network.  An  example  of  this  is  the  IBM/440 
network.  See  Rustin  [1972].  At  any  rate,  obtaining  the  routing  strat- 
egy under  this  assumption  could  prove  extremely  useful  in  the  determina- 
tion of  the  suboptimality  of  certain  decentralized  schemes. 

2 . 6 Previous  Work  on  Optimal  Control  Problems  with  State  Variable 
Inequality  Constraints 

Pioneering  research  into  the  problem  of  the  optimal  control  of 
dynamical  systems  involving  inequality  constraints  on  the'  control  vari- 


ables was  performed  by  Valentine  [1937]  and  McShane  (1939).  Problems 


also  involving  inequality  constraints  on  functions  of  the  state  varia- 
bles alone  were  not  treated  until  the  early  1960 's,  probably  nuDtivated 
by  the  emerging  aerospace  problems  of  that  time.  Pontryagin  [1959] 
provided  the  setting  for  the  new  approaches  with  the  introduction  of 
his  famous  maximum  principle,  a landmark  work  in  the  field  of  optimal 
control.  Gamkrelidze  [1960]  studied  the  situation  in  which  the  first 
time  derivativ  of  the  state  constraint  function  is  an  explicit  function 
of  the  control  variable,  or  so  called  "first  order  state  variable  in- 
equality constraints."  Note  from  equations  (2.12)  and  (2.13)  that  this 
is  our  situation.  This  work  was  devoted  to  finding  necessary  conditions 
in  the  form  of  multiplier  rules  which  must  be  satisfied  by  extremal 
trajectories.  Berkovitz  [1962]  and  Dreyfus  [1962]  derive  similar  re- 
sults from  the  points  of  view  of  the  calculus  of  variations  and  dyneunic 
programming  respectively. 

Subsequent  works  involved  with  necessary  conditions  have  been 
devoted  primarily  to  unravelling  the  technical  difficulties  which  arise 
when  a time  derivative  higher  than  the  first  is  required  to  involve  the 
control  variable  explicitly  (e.g.  Bryson,  Denham  and  Dreyfus  [1963] , 
Speyer  [1968]  and  Jacobson,  Lele  and  Speyer  [1971]).  The  literature  is 
quite  often  at  variance  with  regard  to  the  necessary  conditions  asso- 
ciated with  this  problem,  although  the  later  work  cited  appears  to  pre- 
sent a satisfactory  resolution.  At  any  rate,  we  shall  not  be  concerned 
with  these  differences  as  our  problem  involves  only  first  order  state 


constraints. 


I Computational  aspects  of  the  problem  are  dealt  with  in  Ho  and 

Bretani  [1963]  and  Denham  and  Bryson  [1964] , Both  works  present  itera- 
tive numerical  algorithms  for  the  solution  of  the  open  loop  problem, 
the  procedure  discussed  in  the  latter  being  essentially  an  extension 
of  the  steepest-ascent  method  commonly  used  in  control  problems  uncon- 
strained in  the  state.  As  such,  it  appears  doubtful  that  this  algorithm 
would  exhi.  t acceptable  convergence  properties  for  our  linear  cost 
problem,  particularly  in  the  vicinity  of  the  optimum.  Ho  and  Bretani 
[1963]  report  that  this  is  also  true  of  their  algorithm. 

Little  theoretical  and  computationally  oriented  attention  has  been 
paid  to  the  class  of  control  problems  with  state  variable  inequality 
constraints  and  control  appearing  linearly  in  the  dynamics  and  perfor- 
mance index.  In  this  case,  the  control  is  of  the  bang-bang  type  and 
the  costates  may  be  characterized  by  a high  degree  of  nonuniqueness. 
Maurer  [1975]  examines  the  necessary  conditions  associated  with  this 
problem  when  the  control  and  state  constraint  are  both  scalars,  and 
presents  an  interesting  analogy  between  the  junction  conditions  asso- 
ciated with  state  boundary  arcs  and  singular  control  arcs.  However, 
no  computational  algorithm  is  reported. 

Perhaps  the  most  interesting  computational  approach  presented  for 
the  all  linear  problem  is  the  mathematical  programming  oriented  cutting 
plane  algorithm  of  Kapur  and  Van  Slyke  [1970].  The  basic  algorithm  con- 
sists of  solving  a sequence  of  succeedingly  higher  dimensional  optimal 
control  problems  without  state  space  constraints.  Under  certain  hypo- 
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theses  they  are  ctble  to  prove  strong  convergence  of  the  control  to  the 
optimum.  The  drawbacks  to  this  approach  are  that  the  state  of  the 
augmented  problem  may  grow  unreasonably  large,  and  that  even  uncon- 
strained state  linear  optimal  control  problems  may  be  difficult  to  solve 
efficiently.  In  the  same  paper  Kapur  and  Van  Slyke  [1970]  suggest  for- 
mulating the  problem  as  a large  linear  program  through  discretization 
of  all  constra*..ts , a more  or  less  brute  force  approach  which  is  dis- 
cussed briefly  in  Section  2.7.1. 

Comm  . I to  all  of  the  approaches  described  above  is  that  none 
broach  the  difficult  problem  of  obtaining  feedback  solutions  to  the 
state  constrained  optimal  control  problem.  In  fact,  the  application  of 
necessary  conditions  to  arrive  at  feedback  solutions  is  not  a common 
occurrence  even  for  unconstrained  state  problems.  Notable  exceptions 
are  the  linear  time-optimal  and  linear  quadratic  problem,  for  which  the 
feedback  solutions  are  well  known. 

In  light  of  these  facts,  in  order  to  solve  problem  (2.10) -(2.14)  , 
we  must  first  develop  and  understand  the  necessary  conditions  associated 
with  optimal  solutions  and  creatively  apply  them  to  obtain  a feedback 
solution.  Certainly,  we  must  fully  exploit  the  total  linearity  of  our 
problem,  which  has  not  been  done  heretofore.  As  is  usually  the  case, 
we  shall  be  forced  into  making  even  further  assumptions  cibout  the  prob- 
lem in  order  to  achieve  our  goal.  This  overall  effort  constitutes  the 
primary  mathematical  contribution  of  this  work,  and  is  the  subject  of 


Chapters  J-5. 
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2. 7 Open  Loop  Solutions 

Although  our  foremost  goal  is  the  development  of  feedback  solu- 
tions to  the  optimal  control  problem  of  Section  2.4.,  several  open  loop 
solutions  have  received  consideration.  The  principal  advantage  of 
these  approaches  is  that  they  apply  to  inputs  represented  by  determin- 
istic functions  of  time  of  arbitrary  form.  Also,  at  least  in  principle, 
open- loop  solutions  may  be  implemented  as  feedback  schemes  by  continu- 
ally recalculating  them  in  time  with  the  current  state  taken  as  the 
initial  condition  for  each  problem.  As  such,  open-loop  solutions  are 
worthy  of  brief  mention  at  this  time,  but  we  shall  not  pursue  these 
particular  approaches  further  in  this  thesis. 

2.7.1  Linear  Programming  Through  Discretization 

This  technique  is  a rather  standard  approach  for  linear  optimal 
control  problems.  See,  for  example,  Kapur  and  Van  Slyke  [1970].  We 
first  begin  with  the  assumption  that  the  inputs  are  such  that  the  state 
can  be  driven  to  zero  with  the  available  controls.  Next,  we  select  a 
time  T which  is  sufficiently  large  to  insure  that  T ^ t^,  where  is 
such  that  Jc(t^)  = £.  To  discretize,  we  divide  the  time  interval  [0,  T] 
into  P parts,  each  of  length  At  = tp-tp_j^,  pe  11,  2,  ...,  P).  We  then 
make  the  Cauchy-Euler  approximation  to  the  dynamics 

x(t  ) = [x(t  .)  - x(t  )]/At.  (2.15) 

- p - p+1  - p 


With  this  approximation,  equation  (2.12)  may  be  written 
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x{t  ) = x(t  ) + At[a(t  ) + B u(t  )],  (2.16) 

-p+1  -p  -p p 

V-  p e [0,  1,  P-1] 

and  the  perforroance  index  may  be  approximated  to  first  order  by  the 
discrete  expression 

P 

J , = y;  a'^x(t  ) . (2.17) 

p=0  ^ 

In  this  format  we  have  the  following  constraints: 


D u(t  ) 

p 

< 

C 

(2.18a) 

u(t  ) 

> 

0 

(2.18b) 

- P 

x(t  ) 

> 

0 

(2.18c) 

P 


V-  p e [0,  1,  . ..,  P]  . 

We  now  consider  the  x(t  ) and  u(t)  V-pe  [0,  1,  ....  P]  to  be 

- p - p 

the  decision  variables  of  a linear  programming  problem.  The  total 
number  of  such  variables  is  (n+m)P.  The  non-negativity  constraints 
(2.18b)  and  (2.18c)  are  consistent  with  standard  linear  programming 
format. 

Since  (2.17)  is  a linear  function  of  the  decision  variables,  we 
have  a linear  programming  problem  with  nP  equality  constraints  (2.16) 
and  rP  inequality  constraints  (2.18a) . The  general  format  of  the 
program  is  known  as  the  "staircase  structure",  a form  which  has  re- 
ceived considerable  attention  in  the  programming  literature.  Such 
works  are  devoted  to  exploiting  the  special  structure  of  the  problem  in 


1 

I 


3 


order  to  reduce  computation  through  application  of  decomposition  tech- 
niques. See,  for  example,  Manne  and  Ho  [1974] . Cullum  [1969]  proves 
that  as  the  number  of  discretization  points  P goes  to  infinity,  the 
solution  to  the  discrete  problem  approaches  the  optimal  solution  of  the 
original  problem.  However,  for  a reasonable  size  network  and  for  P 
sufficiently  large  to  insure  good  quality  of  the  approximation,  the  size 
of  the  linear  program  (both  in  terms  of  the  number  of  variables  and  the 
number  of  constraints)  becomes  prohibitively  large  for  practical  appli- 
cation. For  this  reason,  this  ' 'nique  has  been  applied  only  for  the 
purpose  of  obtaining  sample  solutions  to  provide  insight  into  the  pro- 
perties of  optimal  solutions. 

2.7.2  Penalty  Function  Method 

Optimal  control  problems  with  inequality  constraints  on  the  state 
(and/or)  control  have  frequently  been  solved  in  an  open- loop  fashion  by 
converting  them  to  a sequence  of  problems  without  inequality  constraints 
by  means  of  penalty  functions.  One  such  technique  is  presented  by  Las- 
don,  Warren  and  Rice  [1967] . The  penalty  function  detailed  in  that 
paper  works  from  inside  the  constraint,  the  penalty  increasing  as  the 
boundary  is  approached.  Applying  this  technique  to  our  situation,  the 
state  variable  inequality  constrained  problem  (2. 10) -( 2. 14)  is  converted 
to  a problem  without  state  constraints  by  augmenting  the  performance 
index  with  a penalty  function  as  follows: 


-48- 


J 

a 


(x^(t)  + e^/x? (t) ) dt 

i.  j ^ ^ 


> 0, 

i,j  e M,  i yt  2- 


] 

i 

(2.19) 


The  modified  problem  is  then  to  minimize  (2.19)  subject  to  the 
dynamics  (2.1?)  and  the  control  constraints  (2.14). 

Since  the  penalty  function  term 


r 

_ 

1 

i ^ 

Jo  i,j 

(e^/x^(t))dt 

i,j  e W,  i j 

(2.20) 

. 

approaches  infinity  as  any  approaches  its  boundary  x?  = 0,  we  would 
speculate  that  the  minimizing  solution  remains  within  the  constrained 

region  x_  ^ In  Lasdon,  Warren  and  Rice  [1967]  it  is  shown  that  this  ■' 

conjecture  is  true,  and  further  that  the  minimizing  control  as  a func- 
tion of  time  and  the  minimizing  cost  approach  those  for  the  constrained  -J 

problem  as  max  (e^)-*0  i,jeW  i^j.  Also,  since  we  are  approach- 

i,  j ^ ' 

ing  the  constraint  boundary  from  the  interior,  any  solution  for  the 
unconstrained  problem  is  also  feasible  for  the  constrained  problem. 

1 

In  order  to  implement  this  technique,  we  need  to  solve  the  uncon- 
strained problem  by  any  appropriate  numerical  technique  for  successively 
decreasing  values  of  e^.  Gershwin  [1976]  has  created  a program  for  tie 
solution  of  the  penalty  function  approach  to  this  problem  in  which  he 
utilizes  a modified  form  of  differential  dynamic  programming  for  eacli 
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unconstrained  minimization.  The  computational  efficiency  of  the  al-  i 

gorithm  is  greatly  enhanced  by  the  exploitation  of  parametric  linear 
programming  techniques.  Whether  or  not  this  scheme  encounters  numerical 
difficulties  as  grows  very  small  remains  to  be  determined. 

I I 
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Chapter  3 


FEEDBACK  SOLUTION  TO  THE  OPTIMAL 
CONTROL  PROBLEM  WITH  CONSTANT  INPUTS 

3. 1 Introduction 

We  have  formulated  the  data  communication  network  dynamic  message 
routing  problem  as  the  linear  optimal  control  problem  of  Section  2.4. 

In  this  chapte  we  develop  the  fundamental  theory  underlying  a novel 
approach  to  the  synthesis  of  a feedback  solution  to  that  problem.  The 
technique  is  predicated  upon  the  assumption  that  all  inputs  to  the  net- 
work are  constant  in  time.  In  this  section  we  present  a brief  review 
of  the  development. 

We  begin  by  presenting  the  necessary  conditions  for  the  general 
deterministic  problem  (inputs  not  constrained  to  be  constant)  and  dis- 
cuss these  conditions  in  some  detail.  We  then  show  that  these  condi- 
tions are  also  sufficient,  a most  fortuitous  situation  since  it  guaran- 
tees the  optimality  of  trajectories  which  we  eventually  shall  construct 
using  these  conditions. 

The  subsequent  discussion  is  restricted  to  the  constant  inputs 
case.  Based  exclusively  on  the  necessary  and  sufficiency  conditions, 
a geometrical  characterization  of  the  feedback  space  for  constant  inputs 
is  presented:  the  regions  of  ^-space  over  which  the  same  controls  and 
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sequence  of  states  on  and  off  boundary  arcs  are  optimal  are  convex  poly- 
hedral cones  in  the  state  space.  It  is  this  special  property  (a  result 
of  the  total  linearity  of  the  problem)  which  embodies  the  ade^tability 
of  the  constant  input  formulation  to  the  synthesis  of  a feedback  solu- 
tion. What  one  needs  is  an  algorithm  to  specify  these  regions  and  the 
appropriate  controls . 

Next,  we  present  several  simple  examples  to  illustrate  how  these 
regions  may  be  constructed.  This  is  accomplished  by  producing  a certain 
set  of  optimal  trajectories  backward  in  time  from  the  final  state  = O.- 
Each  portion  of  these  trajectories  generates  one  of  these  convex  poly- 
hedral conical  regions,  thereby  providing  the  optimal  control  in  that 
region.  If  a sufficiently  complete  set  of  trajectories  is  calculated, 
we  manage  to  find  enough  regions  to  fill  out  the  entire  space. 

The  remainder  of  the  chapter  is  devoted  to  the  extension  of  this 
concept  to  the  general  network  optimal  control  problem  with  constant 
inputs.  Taking  the  lead  from  the  examples  performed,  an  algor- 
ithm is  presented  for  the  construction  of  the  conical  regions  from  a 
comprehensive  set  of  backward  optimal  trajectories.  This  algorithm  is 
in  general  form,  and  several  specific  issues  are  raised  regarding  its 
execution.  An  issue  of  central  importance  is  the  determination  of  the 
appropriate  set  of  costates  required  by  the  algorithm.  The  resolution 
of  this  question  is  essentially  geometrical  in  nature,  relying  upon  the 
interpretation  of  the  Hamiltonian  as  a continuously  rotating  hyperplane. 
In  fact,  many  of  the  arguments  are  geometrical,  so  that  we  frequently 
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use  two  and  three  dimensional  examples  for  illustration  of  the  ideas. 

Unfortunately,  there  are  several  characteristics  associated  with 
the  backward  construction  method  which  complicate  its  formulation  as  a 
compact  computational  algorithm  for  the  general  network  model.  These 
properties  are  introduced  by  way  of  examples  in  the  Section  3.3.4. 
Looking  ediead,  however,  we  are  able  to  show  in  Chapter  4 that  these 
bothersome  pr  nerties  are  non-existent  in  the  case  of  problems  involving 
single  destination  networks  with  all  unity  weightings  in  the  cost  func- 
tional. 

3.2  Feedback  Solution  Fundamentals 

3.2.1  Necessary  and  Sufficiency  Conditions 
The  necessary  and  sufficiency  conditions  to  be  presented  are  valid 
for  arbitrary  deterministic  inputs.  For  ease  of  reference,  we  restate 
our  problem  here. 


Minimization 

min  J(x(t)) 
u(t) eU 

(3.1) 

Performance  Index 

J(x(t))  = 1 ^ 

[o^x(t) ]dt 

(3.2) 

Dynamics 

x(t)  = B u(t)  + 

ait) 

(3.3) 

Boundary  Conditions 

x(t^)  = 

0 

(3.4) 

State  Constraints 

x(t)  2 0 t 

e (tQ, 

^f> 

(3.5) 

r 
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f 

I 

ID  u(t)  < C i 

V-  t e [t^,  t,l  (3.6) 
u(t)  >0  ^ ^ 

■ Theorem  3.1  (Necessary  Conditions) 

Let  the  scalar  functional  h be  defined  as  follows 

, h(u(t) , ^(t) ) = V^(t)x(t)  = V^(t) [B  u(t)  + a(t) ) . (3.7) 

A necessary  condition  for  the  control  law  u*(')  e (/  to  be  optimal 
for  problem  (3.1) -(3.6)  is  that  it  minimize  h pointwise  in  time,  naunely 

V^(t)B  u*(t)  < V^(t)B  u(t)  (3.8) 

•!  V-  u(t)  £ U V-  t £ Uq,  t^J . 

The  costate  ^(t)  is  possibly  a discontinuous  function  which  satisfies 
the  following  differential  equation 

-dX(t)  = adt  + dn (t) , t £ (tg,  t^]  (3.9) 

where  componentwise  di]_(T)  satisfies  the  following  complementary  slack- 
ness condition 

I 

I j j 

X-/ (t)  dn^  (t)  = 0 \ V t e [t,  t,l  (3.10) 

t 1 / Of 

* dn^  (t)  < 0 ‘ i,  j E W,  j i.  (3.11) 

The  terminal  boundary  condition  for  the  costate  differential  equation 
is 

^(tf)  = y free  (3.12) 


and  the  transversal ity  condition  is 
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^ (t^)x(t^)  = 

Finally,  the  function  h is  everywhere  continuous,  i.e. 

h(u(t  ),  ^(t  ))  = h(u(t^),  ^(t^)) 

V-  t e [tp,  t^l  . 


(3.13) 


(3.14) 


Proof: 

Jacobson,  Lele  and  Speyer  [1971]  present  a generalized  Kuhn-Tucker 
theorem  in  a Banach  space  for  the  minimization  of  a differentiable  func- 
tion subject  to  inequality  constraints.  For  our  problem,  it  calls  for 
the  formation  of  the  Lagrangian 


(T)dT  + 


pf  T 

= 1 ax 

+ r ^ dn(T)\ 


(T) [B  U(T)  + a(T)  - X(T) ]dT 


(T)  + V X(t^) 


(3.15) 


where  is  an  n 1 vector  adjoining  the  state  constraints  which  satis- 
fies the  complementary  slackness  condition 


(T)x(T) 


= 0 


(3.16) 


dri(T)  ^ 0 


V-  T c 


[t 


0' 


(3.17) 


The  vector  V which  adjoins  the  final  condition  is  an  n x 1 vector  of 


arbitrary  constants. 
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For  u^*(’)  to  be  optimal,  J must  be  minimized  at  u*(*)  where  x ( • ) , 
x(t^)  and  tg  are  unconstrained  and  u is  constrained  by  u e U,  Taking 
the  differential  of  J with  respect  to  arbitrary  variations  of  x ( ' ) , 


3^(t^)  and  t^  we  obtain 


dJ 


^6x(T)dT  + a^x(t^)dt^ 


-r^ 


(T)6x(T)dT  + 

0 ■ ■''0 


I '^n 


(t)6x(t) 


+ V dx(tg) 


(3.18) 


where  6x  is  the  variation  in  x for  time  held  fixed  and 


dx(t^)  = 6x(t^)  + x(t^)dt^ 


(3.19) 


is  the  total  differential  of  x(t^).  We  next  integrate  the  third  term 
of  (3.18)  by  parts  and  rearrange  to  obtain 


dJ  = I 6x*(t)  [a_dx  + dr]_(T)  + d^(T>] 
'0 

- x’’(t^)6x(tj)  + X^(to)6x(tQ) 


+ a'^x(t^)dtj  + v^dx(tj) 


(3.20) 


Since  the  initial  conditions  are  fixed  we  liave  63c(tQ)  = Substituting 


for  d5c(t^)  from  (3.19): 


dJ 


(t) (adT  + dn(x)  + dX(T)}  + 


d 


I 
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+ [\^  - X'^(t^)]6x(t^) 

+ [a^x(t^)  + v"^x(tg)  ]dtg. 


(3.21) 


Now,  in  order  for  J to  be  stationary  with  respect  to  the  free  variations 


6x(t) , 6x(t^)  and  dt^  we  must  have 


adT  + dn(T)  + dX(T)  = 0 


^(tf)  = v_  ))_  free 
^"^x(tf)  = ^^(t^)x(t^)  = -a^x(t^) 


(3.22) 

(3.23) 

(3.24) 


Equations  (3.16)  and  (3.17)  together  with  the  constraint  x > 0 imply 

dn^(t)x?(t)  =0  V- T e (t  , t ) (3.25) 

11  Of 

i,j  e W,  j i. 

:(T)dT  by  parts  in  equation 


If  we  integrate  the  term 


X'^(T)X( 

•'"o 


(3.15)  and  substitute  equations  (3.16)  and  (3 . 22) - ( 3.  24)  into  (3.15) 
we  obtain 

rt 

,T 


J = I ^'(t)[B  u(t)  + a(T))dT. 
'0 


(3.26) 


In  order  for  J to  be  minimized  with  respect  to  ^(  •)£((,  the  term 
T 

X (t)  B u(t)  must  clearly  be  minimized  pointwise  in  time,  that  is 


a"^(t)B  u*(t)  < X^(t)B  u(t) 


V-  u(t)  £ LI  , t c (tp,  tj]  . 


(3.27) 
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Thus,  we  have  accounted  for  equation  (3.8),  leaving  only  (3.14)  to  be 
proven. 

To  this  end,  let  us  assume  that  we  have  an  optimal  state  trajec- 
tory x*(t)  and  associated  costate  trajectory  ^(t) , t e [t^,  . Then 

by  the  principle  of  optimality,  for  any  fixed  T ^ t^,  the  functions 
3{*(t)  and  ^(t) , t £ [t^,  t] , are  optimal  state  and  costate  trajectories 
which  cari^  the  state  from  to  x(t^)  = x(T) . Hence,  all  of  our  pre- 
vious conditions  apply  on  [t^,  T]  with  x(t^)  = x(T) . Applying  the 
transversality  condition  (3.24)  uL  t^  = T,  we  oV-i-ain 

V^(T)x(T)  = -o^x(T) . (3.28) 

Since  equation  (3,28)  holds  for  all  T e [t^,  t^I  and  x(T)  is  everywhere 
T 

continuous,  then  ^ (T)x(T)  must  be  everywhere  continuous.  This  proves 
equation  (3,14).  ■ Theorem  3.1 

All  of  the  necessary  conditions  which  refer  to  the  functional  h 
defined  in  (3.7)  also  apply  to  the  more  general  (and  familiar)  func- 
tional called  the  Hamiltonian  defined  as 

H(x(t)  , u(t)  , ^(t),  i^(t))  = ot^x(t)  + ^^(t)x(t)  4 ]j^(t)  Jc(t).  (3.29) 

The  vector  of  multiplier  functions  jj(t)  is  defined  componentwise  as 

dn?(t) 

M^t)  = - (3.30) 

1 dt 

and  therefore  p?  is  defined  only  at  points  for  which  ri?  is  absolutely 
continuous.  We  need  not  be  concerned  with  this  technicality  when  we 
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use  h,  which  still  embodies  the  important  properties  of  H for  our 
problem. 

For  emphasis,  we  shall  discuss  next  the  nature  of  the  costates  as 
functions  of  the  corresponding  states  as  determined  by  equations  (3.9)- 
(3,10).  At  points  of  absolute  continuity  of  the  associated  costate 
differential  equation  may  be  expressed  in  the  familiar  form 


1 


^JL 


+ m|(t)  , 


(3,31) 


when  the  s .ate  x^  > 0 (x^  is  said  to  be  on  an  interior  arc)  then  equa- 

1 1 


tion  (3.10)  implies  dn^  = 0,  and  equation  (3.30)  gives  = 0.  There- 


fore,  equation  (3.31)  reduces  to 


(T) 

1 


= when  x^ (T)  >0. 

1 1 


(3.32) 


When  the  state  x|  = 0 (x?  is  said  to  be  on  a boundary  arc) , its  costate 
is  possibly  discontinuous,  depending  on  the  nature  of  n?.  At  points 
for  which  r|^  is  absolutely  continuous,  we  have  from  equations  (3.10) 
and  (3.11)  that  dn^  0 (since  x^  = 0) , and  therefore  equation  (3.30) 


implies  M?(t)  £ 0.  Hence, 


(T)  = (T) 

1 11 


Vi^(T)  ^ 0 


when  x.(t)  = 0. 
1 


(3.33) 


(3.34) 


On 


the  other  hand  at  times  i when  n?  experiences  jumps  of  magni- 


tude An:^(T  ),  equation  (3.11)  indicates 
1 P 
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An?(T  ) < 0 (3.35) 

1 p - 

and  from  equation  (3.9)  we  have 


AX^'(t  ) = X?(t''')  - X^(t”)  = -An^(T  ) > 0. 

ip  xp  ip  ip- 


(3.36) 


That  is,  jumps  in  the  costate  are  always  positive.  The  above  situations 
are  depicted  in  Figure  3.1  for  a possible  state-oostate  trajectory  pair. 

The  times  at  which  the  state  enters  or  exits  from  a boundary  arc 
are  called  boundary  junctions.  Note  that  the  minimum  value  which  the 
slope  of  uie  costate  may  attain  on  or  off  a boundary  arc  is  Aside 

from  jumps,  a characteristic  of  the  costate  which  distinguishes  the 
state  constrained  problem  is  the  possible  non-uniqueness  of  the  costate 
for  a given  optimal  trajectory.  As  such,  it  is  clear  that  the  costates 
may  not  be  interpreted  as  the  partial  derivative  of  the  optimal  cost 
with  respect  to  the  state,  commonly  known  as  the  influence  function. 
Furthermore,  the  non-uniqueness  of  the  costate  presents  special  impli- 
cations for  the  feedback  scheme  to  be  developed.  An  example  of  this 
behavior  and  a discussion  of  its  significance  is  presented  in  Section 
3. 3.4.2 . 

In  general,  any  trajectory  obtained  from  a set  of  necessary  condi- 
tions is  an  extremal,  and  as  such  is  merely  a ceindidate  for  an  optimal 
trajectory.  Fortunately,  in  our  problem  it  turns  out  that  any  such 
extremal  trajectory  is  actually  optimal,  as  is  shown  in  the  following 


] 





theorem. 
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Theorem  3.2 


The  necessary  conditions  of  Theorem  3.1  are  sufficient. 


Proof: 


Let  x*(t),  ja*(t),  ^(t)  , r[(t)  satisfy  the  equations  (3.2)-  (3.6) 
and  the  necessary  conditions  (3. 7)- (3.14)  of  Theorem  3.1.  Also,  let 
x(t) , u(t)  be  any  state  and  costate  trajectory  satisfying  (3.2) -(3.6) 


6J  = J(x)  - J(x*) 


T , V , 

= I a (t)  (x 


(t)  - x*(t)  ) dr 


■/.: 


^ (-dV^(T)  - dQ_'^(T))  (X(T)  - X*(T)) 


by  substituting  from  (3.9)  and  expanding  obtain 


;'^(T)dA(T)  - x'^(T)dn(T)  + X*'^(T)dX(T)  + x*'^(T)dn(T)) 


From  equation  (3.16) 


* (T)dn(T)  = 0.  We  now  integrate  the  first 


and  third  term  in  6J  by  parts: 


/*^f  T 
I X (T)dX 


(t)  = l^(t^)x(t^)  - ^^(tQ)jc(tQ) 


'f  rj, 

X (T)x(T)dT 


J - 


(T)dX(T)  = A‘(t^)x  (t^)  - A (tg) 


- £‘ 


(T)dT. 


Now , by  (3.4) 


x(tg)  = X (^q)  = 3^  and 


x(t^)  = x*(t^)  = £. 


Substituting  these  expressions  we  obtain 


= f ^ a^(T)x 

K 

r^f  T 

- J X (T)dn(T) 


(T  ) - X (T  )X*  (t)  ) dT 


= r ^ V^(T)B(U' 

- I xNT)dn(T), 


(t)  - U*(T))dT 


But  by  (3.8) 


C^f  T 

J X (t)b(u 


(t)  - u*  (t)  ) dT  > 0 


and  since  x(t)  >0  and  dri(T)  < 0 we  have 


(T)dri(T)  £ 0. 
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Therefore  6j  ^ 0 ¥ ^( • ) e U,  x ( • ) ^ 0.  ■ Theorem  3.2 

Although  it  is  not  immediately  apparent  how  one  may  utilize  the 
necessary  conditions  to  obtain  optimal  solutions  (this  if  most  often 
the  case) , certain  fundamental  characteristics  of  optimal  solutions  may 
be  deduced  immediately  from  the  form  of  these  conditions.  We  begin  this 
discussion  by  noting  that  finding  the  optimal  control  function  u*(0 
reduces  from  (3.8)  to  solving  at  every  time  T e [t^,  t^]  the  linear 
program  with  decision  vector  u(t^ : 

u*(T)  = ARG  MIN  [^'^(T)x(T)] 
u(T) cU 

= ARG  MIN  [^"^(DB  u(T)  ] (3.37) 

u(T) eU 

!D  u < C 
u > 0 

This  is  a fortuitous  situation,  since  much  is  known  about  characterizing 
and  finding  solutions  of  linear  programs.  We  know,  for  instance,  that 
optimal  solutions  always  lie  on  the  boundary  of  the  convex  polyhedral 
constraint  region  U. 

We  now  proceed  to  represent  the  solution  to  the  linear  program 
(3.37)  for  the  specific  form  of  the  matrices  B and  D which  correspond 
to  our  network  problem.  The  minimization  can  actually  be  performed  on 
one  link  at  a time.  Consider  the  link  (i,k)  and  a possible  set  of 


associated  controls 


u , u , .. 

ik  ik 


i-1  i+1 

^ik  ' '^ik  ' 


•'  ^k- 


A given  control  may  appear  in  one  of.  the  two  following  ways: 


1)  enters  into  exactly  two  state  equations : 


X?  (t)  = -u?  (t)  + . . . + a^ 
1 ik  1 


xj(t)  = +u3j^(t)  + 


(3.38) 


2)  enters  into  exactly  one  state  equation: 


.k , , k k 

X . (t)  = -u . , + ...  + a . 

Ilk  1 


(3.39) 


Hence,  all  controls  on  link  (i,k)  contribute  the  following  terms  to 


a'^B  u: 


aht)  - A^(t))u^.  (t) 

k 1 ik 


(A^(t)  - A^(t))u^,.(t) 

K 1 X A 


+ (A^“^(t)-A^"^(t))u:"^(t) 
k 1 ik 


(3.40) 


+ (A^'^^(t)-A^'^^(t))u^^^(t) 
k 1 ik 


N N N 

+ (A"(t)  - A (t))u"  (t) 

k 1 ik 


where  Aj^(t)  = 0.  Equations  (2.4),  (3.37)  and  (3.40)  determine 


w 


-es- 


The  optimal  control  law  at  time  t on  (i,k); 

• “ik"'  ■ 0 ''  j '' ' 

if  (xSt)  - x*'(t))  < (X^(t)  - x3(t))  V-  j / il 

k 1 K X 

and  (X^(t)  - X^(t))  0. 


ik 


Ik' 


(3.42) 


u^.  (t)  = 0 V-  j ji!  [«.  . . . ,m] 

XK 


if  (xSt)  - xSt))  = (x^^^(t)  - x^-^^t))  = ...  = 

k 1 k 1 

= (X™(t)  -X^(t))  < (X^(t)  - xj^(t)) 

V-  j g [£  , 2.+1 , . . . , m] 

and  (X^(t)  - xSt))  = (X^^^t)-X^^^t))  = ...  = 

= (X^{t)  - X^(t))  < 0. 


• 4‘'>  " - •••  * “Tk"’  2-" 


ik 


(3.43) 


(t)  = 0 V-  j . ,ml 

ik 

if  (xJ(t)-xSt))  = (Xj^^t)  - X^'"^(t))  = ...  = 

k X K X 

= (xf(t)  - x'l'(t))  = 0 

k 1 


and  (X^(t)  - X^(t))  > 0 

k X 


4 


V j t [il,  K.+  1,  . . . , ml. 
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The  actual  computation  of  the  optimal  control  at  time  t requires 
knowledge  of  ^(t) , which  in  turn  requires  knowledge  of  the  optimal  state 
for  time  greater  than  or  equal  to  t.  This  is  the  central  difficulty  in 
the  application  of  necessary  conditions  in  the  determination  of  a feed- 
back solution.  The  remainder  of  this  thesis  is  essentially  devoted  to 
demonstrating  how  this  difficulty  may  be  surmounted  for  our  problem. 

However,  may  immediately  deduce  several  properties  of  the  opti- 
mal control  from  examining  the  general  form  of  (3.41) - (3.43) . It  is 
evident  that  the  quantities  which  determine  the  optimal  controls  are  the 
coefficients  of  the  form  (X^(t)  - A^(t))  which  multiply  the  control 
u^k^t).  The  only  situation  under  which  it  is  ever  optimal  to  have  u|j^(t) 
strictly  positive  is  if  (A^(t)  - A|(t))  ^ 0.  In  words,  this  condition 
says  that  it  is  optimal  to  send  messages  with  destination  j from  node  i 
to  node  k at  time  t only  if  the  costate  associated  with  x?(t)  is  greater 
than  or  equal  to  that  associated  with  x^(t).  This  fact  gives  rise  to  a 
very  interesting  analogy:  The  flow  of  messages  in  a data  communications 

network  can  be  roughly  visualized  as  the  frictionless  flow  of  fluids  in 
a network  of  pipes  with  storage  at  tanks  in  the  nodes.  We  may  then 
relate  the  costates  ^^(t)  and  A^(t)  to  the  pressures  at  the  storage 
areas  x^  and  x^  respectively,  and  the  difference  (A^(t)  - A^(t))  to  the 
associated  pressure  difference.  In  this  vein,  we  shall  refer  to 
(A^(t)  - A^(t))  as  the  costate  difference  which  exists  at  time  t between 
node  i and  node  k and  is  associated  with  the  traffic  going  to  destina- 


I 


tion  j.  It  is  therefore  optimal  to  send  messages  of  a given  destination 
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from  one  node  to  an  adjacent  node  only  if  the  costate  difference  is  nega- 
tive or  zero. 

According  to  equation  (3.41) , if  the  costate  difference  associated 
with  destination  j on  link  (i,k)  is  strictly  negative  and  less  than  the 
remaining  costate  differences  on  this  link,  then  the  optimal  control  is 
u^j^(t)  = and  all  other  controls  are  zero.  However,  equations  (3.42) 
and  (3,43)  ..eveal  that  when  two  or  more  costate  differences  on  the  same 
link  are  non-positive  and  equal,  the  associated  optimal  control  will  not 
be  uniquely  determined.  In  these  situations,  the  optimal  solution  set  is 
in  fact  infinitely  large.  Such  non-uniqueness  is  a fundamental  property 
of  linear  programs. 

3.2,2  Controllability  to  Zero  for  Constant  Inputs 

Henceforth  we  shall  be  considering  only  the  situation  in  which  all 
the  inputs  are  constant  functions  of  time  over  the  interval  of  interest 
t e [tQ,  tf] . The  utility  of  this  assumption  for  the  representation  of 
data  network  operation  is  discussed  in  Section  5.1.  In  this  section, 
we  present  a simple  theorem  which  characterizes  all  those  inputs  which 
allow  the  state  to  be  driven  to  zero  under  given  link  capacity  cons- 
straints.  We  begin  with  a definition: 

Definition  3.1:  We  denote  by 

,>JT  = {){|  = B.  u and  ^ e U}  i (3.44) 

the  set  of  feasible  flows  attainable  through  the  available  controls. 
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The  reason  for  the  negative  sign  in  the  definition  of;^’is  basically 
one  of  notational  convenience.  We  shall  also  refer  to.:^'as  the  x-con- 
straint  figure.  Note  that  since  ti  is  a bounded  convex  polyhedron  in 
R"*  and  B is  a linear  mapping  from  R*”  to  R"^,  then  is  a bounded  con- 
vex polyhderon  in  R^. 

Theorem  3.3  (^"'introllability  to  zero,  constant  inputs) 

All  initial  conditions  of  the  system  (3.3)-(3,6)  are  controllable 
to  zero  under  constant  inputs  if  and  only  if 

a£Int(,i/-)  (a  e R‘^,J  c R*^) 

where  denotes  the  interior  of  the  x-constraint  figure. 

Proof:  Suppose  a e Int (.'/). 


Figure  3. 2 


x-Constraint  Figure 
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We  shall  proceed  by  showing  that  it  is  possible  to  pick  a control 
corresponding  to  a feasible  flow  which  simultaneously  brings  all  compon- 
ents of  any  initial  state  x(0)  to  zero. 

There  is  a neighborhood  N(a,  e)  which  lies  entirely  in  Inti.'Jt')  for 
e > 0 sufficiently  small.  See  Figure  3.2.  Consequently,  we  may  pick 

a constant  control  u'  such  that  for  all 

— 1 


-b^u'  = a^  + 


x?(0) 

1 

|x(0) I 


where  b|^  is  the  row  of  corresponding  to  x|,  and  x^(0)  is  the  corres- 
ponding component  of  x(0) . The  norm  is  in  the  Euclidean  sense.  Apply- 
ing the  control  u'  we  obtain 

x^(0) 

x?  (t)  = b^u'  + a:  = -c  

1 —1—  1 


x(0) 


since  e > 0,  all  components  of  %(Q)  are  brought  to  zero  in  the  time 

|x(0) 

e 

On  the  other  hand,  suppose  that  a ^ Intl'Jt  ) . Then  a falls  either 

• • n 

on  a boundary  of.i^'or  in  the  complement  of.^^'with  respect  to  R . It 

now  must  be  shown  that  in  either  case  there  exists  some  initial  condi- 
tion which  is  i.ot  controllable  to  zero. 

We  begin  with  the  case  in  which  £ falls  at  a boundary  point  of,'/'; 
that  is,  a falls  on  a face  of  the  convex  polyhedron  As  a prelimin- 

ary, we  recognize  the  fact  that  since  all  of  the  components  of  ^ are 


x^(0) 


T = 
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non-negative,  then  the  face  of  .4"  on  which  it  falls  lies  at  least  par- 
tially in  the  non-negative  orthant  of  It  is  not  difficult  to  see 

that  any  face  of  which  lies  in  the  non-negative  orthant  of  R^  has 
the  following  properties: 

(i)  Consider  the  hyperplane  which  contains  the  face.  Then  the 
coefficients  of  the  equation  of  the  hyperplane  are  all  non-negative; 
that  is,  the  ’"'.it  normal  vector  to  the  face  has  all  non-negative  com- 
ponents. 

(ii)  At  least  one  of  the  components  of  the  unit  normal  vector 
is  strictly  positive. 

Let  n denote  the  unit  normal  vector  to  the  particular  face  of 
on  which  a lies.  See  Figure  3.3. 


Figure  ?.3 


Attainabie  Fl<iws  in  the  Tiest-nce 
of  Constant  Inputs 
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Now,  since  -x  = -B  u - a it  is  evident  from  Figure  3.3  that  a 
given  flow  -x  can  be  realized  by  a feasible  flow  u with  inputs  a 
if  and  only  if 

-X  • n < 0,  (3.45) 

We  now  proceed  to  show  that  if  a lies  at  a boundary  point  of 
then  we  can  always  find  some  initial  condition  which  cannot  be  brought 
to  zero  with  the  available  controls.  To  this  end,  consider  the  follow- 
ing subset  of  R*': 

= {x  e R^'lx^  = 0 if  n^  > o} 

where  n^  is  the  component  of  n corresponding  to  x^.  Now,  choose  any 
initial  condition  such  that  ip.  Clearly  such  a point  always 

exists.  We  now  shall  show  by  contradiction  that  any  such  initial  condi- 
tion can  never  be  brought  to  the  final  state  ~ Suppose  that 

the  state  can  be  brought  to  zero  at  some  time  t^.  Then  the  state  must 
hit  at  some  time  T'  t^.  Therefore,  at  some  time  T"  < T'  we  must 

’£  a 

have  Xj^(t")  < 0 for  at  least  one  component  x^  of  the  state  vector  such 
£ 

that  > 0.  The  existence  of  such  a component  is  guaranteed  by 
property  (ii)  above.  Also,  all  of  the  other  components  of  >c  associated 
with  positive  components  of  ri  must  be  such  that  they  have  non-positive 
time  derivatives  at  T".  Hence,  since  by  (i)  above  all  components  of  ri 
are  non-negative  we  have 

-x(t")  • n = V,  -x^(T")n^  > 0. 

i.jtSJ  ^ ^ 
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The  above  inequality  contradicts  (3.45)  eind  therefore  any  x(tp)  t 'i' 
cannot  be  brought  to  zero. 

Finally,  consider  the  case  in  which  a e ^ Then  it  is  easily 
seen  that  there  is  at  least  one  component  of  the  state  which  increases 
without  bound  for  any  u e U.  Therefore,  no  initial  condition  is  con- 
trollable to  zero  in  this  case  and  the  theorem  is  proved. 

■ Theorem  3.3. 

Controllability  to  zero  is  certainly  a necessary  condition  for 
the  existence  of  an  optimal  solution  to  (3.1)-(3.6),  although  not  in 
itself  sufficient.  However,  as  our  feedback  synthesis  technique  will 
be  based  upon  trajectories  which  satisfy  the  necessary  conditions,  we 
are  guaranteed  of  their  optimality  by  the  sufficiency  Theorem  3.2. 

We  shall  therefore  have  no  need  of  an  explicit  proof  of  the  existence 
of  an  optimizing  control. 

3.2.3  Geometrical  Characterization  of  the  Feedback  Space  for 
Constant  Inputs 

In  what  follows,  we  shall  assume  that  the  controllability  to  zero 
condition  of  Theorem  3.3  is  satisfied.  The  following  is  a consequence 
of  the  optimal  control  characterization  (3.41)-  3.43),  and  as  such  is 

a corollary  to  Theorem  3.1. 

Corollary  3.1  If  £ - constant,  then  there  always  exists  a solution  to 
(3.1) -(3.6)  with  controls  piecewise  constant  in  time  and  trajectory 
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with  piecewise  constant  slopes. 

Proof; 

For  a given  costate  trajectory  ^(t) , t e [t^/  t^] , the  optimal 
control  is  given  by  (3.41)- (3,43) . For  those  controls  with  unique 
optimal  values,  the  determining  factor  is  the  sign  and  relative  magni- 
tude of  the  associated  costate  difference.  Over  periods  of  time  for 
which  these  factors  remain  the  same,  the  optimal  control  is  constant 

£ J,  £ 

and  is  given  by  u.,  = C,,  or  u..  = 0.  Those  controls  not  uniquely 

IK  iK  1 K 

determined  over  certain  periods  of  time  (as  in  situations  (3.42)  and 
(3.43))  may  achieve  any  values  which  satisfy  certain  constraints,  such 

as  — ^ik"  only  additional  requirement  on 

j=d  j=£. 

these  controls  may  be  that  they  maintain  certain  states  on  boundary  arcs 
over  certain  periods  of  time.  Since  the  inputs  are  constant,  any  such 
requirement  may  always  be  satisfied  by  constant  valued  controls  over 
the  appropriate  time  period.  As  the  controls  have  been  shown  to  be 
piecewise  constant  and  the  inputs  are  constant,  it  follows  immediately 
from  equation  (3.3)  that  the  associated  optimal  trajectories  have  piece- 
wise  constant  slopes.  ■ Corollary  3.1 

We  note  in  passing  that  the  above  corollary  also  holds  for  piece- 
wise  constant  inputs. 

The  solution  to  the  constant  input  problem  is  of  the  bang-bang 


variety  in  that  the  optimal  control  switches  intermittently  among 


1 
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boundary  points  of  U,  Also,  in  situations  when  one  or  more  costate 
differences  are  zero  or  several  are  negative  and  equal,  the  control  is 
termed  singular.  Under  such  circumstances,  the  optimal  control  is  not 
determined  uniquely  from  the  laws  (3.41) -(3.43) . In  the  solution  tech- 
nique to  be  presented,  this  non-uniqueness  will  play  a major  role. 

Owing  to  the  bang-bang  nature  of  the  control,  every  optimal  tra- 
jectory may  ) - characterized  by  a finite  number  of  parameters.  We  now 
present  a compact  set  of  notation  for  specifying  these  parameters. 

Definition  3.2 

U(x)  = {Uq,  u^,  u^,  ...»  (3.46) 

and  T(2C^)  = {t^,  t^^,  ...,  t^} 

are  a sequence  of  optimal  controls  and  associated  control  switch  time 
sequence  which  bring  the  state  x optimally  to  0^  on  t e [t^,  t^] , where 
is  the  optimal  control  on  t e [t^,  t^^^) , p e [0,  1,  ...,  f-1] . 

When  dealing  with  the  necessary  conditions,  an  additional  pro- 
! perty  of  a given  trajectory  which  shall  be  of  interest  is  which  states 

travel  on  boundary  arcs  and  over  what  periods  of  time.  This  informa- 
tion is  summarized  in  the  following  definitions: 

Definition  3.3 

B = {x^|x^(T)  =0  V T E [t  , t ^-)}  (3.47) 

p 1 1 p p+ 1 

is  the  set  of  state  variables  travelling  on  boundary  arcs  during  tiie 
application  of  a . 


J 
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Definltion  3,4 

B(x)  = {8q,  8^,  ....  8^_^}  (3.48) 

is  the  sequence  of  sets  8 corresponding  to  the  application  of  U(x)  on 
T(x).  8(x)  is  referred  to  as  the  boundary  sequence. 

In  the  forthcoming  development,  we  shall  be  interested  in  regions 
of  the  state  space  which  have  the  following  property;  when  we  consider 
every  point  of  the  region  to  be  an  initial  condition  of  the  optimal  con- 
trol problem,  a common  optimal  control  sequence  and  a common  associated 
boundary  sequence  apply  to  all  points.  Formally,  we  define  the  follow- 
ing subset  of 

Definition  3.5 

8)  = {x  e R^|u(x)  = U and  = 8}.  (3.49) 

Note  that  since  any  two  distinct  members  of  ?<^(U,  8)  have  associated  with 
them  the  same  sequence  of  optimal  controls,  the  number  of  switch  times 
in  their  respective  sequences  is  the  same.  However,  the  values  of  the 
particular  switch  times  will  in  general  be  different.  Suppose 
x^  e 8)  and  e B)  , Then  the  switching  time  sequences  are 

distinguished  with  the  following  notation: 


Notation  3.1 


T(Xj^) 

T(x2) 


{t’,  t|, 

{t^,  t^. 


t^} 


• • • » 


(3.50) 


The  above  definitions  are  illustrated  pictorially  in  Figure  3,4. 

It  may  appear  at  first  that  the  regions  8)  defined  above  are 

too  special  to  be  of  any  utility.  However,  we  shall  soon  demonstrate 
that  such  regions  are  related  to  the  basic  building  blocks  of  the  feed- 
back space.  With  this  as  motivation,  we  begin  by  providing  a fundaunen- 
tal  geometric  property  of  these  regions. 

Theorem  3 . 4 (Geometrical  characterization) 

Let  the  inputs  a = constant.  Let  U and  8 be  control  and  boundary 
sequences  respectively  as  defined  in  ( 3.46)  - ( 3. 48)  . Also,  let  Vf'(U,  8) 
be  the  subset  of  R*'  as  defined  in  (3.49).  Then  V®’(CJ,  8)  is  a convex 
polyhedral  cone  in 

Proof : The  basic  elements  are  embodied  in  the  following  lemmas: 

Lemma  3.1.  Let  the  inputs  a = constant.  Suppose  e R^  and  ^ R^. 

If  X2  = Y is  a scalar  Y ^ 0/  then 

U(X2)  = U(x^) 

and  S(2i2^  “ 

Proof  of  Lemma  3.1: 

For  the  purpose  of  this  proof,  we  shall  consider  time  to  be  pro- 


pagating backward  from  t^. 


a costate  history  A_(t)  for  which  U(x, ) is  an  optimal  control  sequence 
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which  brings  ^ £.•  These  sets  are 

TU^)  = t;-,  ...,  t^}  (3.53) 

w^ere  t"  = Yt'  (3,54) 

P P 

p E [0,  1,  ...f] 

d^2(T)  = + <^112  (3.55) 

where  ~ "l^Ai  (3.56) 

and  dn  /-^'  -r  ydrij^ (Vy)  • (3,57) 


To  show  that  indeed  this  set  optimally  brings  jc^  to  £ we  have  to 
show  the  following  assertions: 

(i)  The  application  of  U(Xj^)  on  T(x_2>  drives  the  state 
from  £ to  x^  = YXj^  backWw.rd  in  time. 

(ii)  8(x^2^  = • 

(iii)  £2  end  satisfy  all  of  the  necessary  conditions. 

(iv)  b(Xj^)  is  optimal  on  T(x2). 

Step  (i) . 

Since  our  system  is  time  invariant,  for  convenience  we  set 

t'  = t"  = t,  = 0 as  the  initial  time  in  the  bac)cward  sense.  Then  if 
f f f 

we  denote  by  Xj^(-)  and  state  trajectories  resulting  from  the 

apllication  of  U(x^)  on  T(Xj^)  and  T(x2)  respectively,  integration  of 
the  state  dynamics  backward  in  time  yields 
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^2 


(T/Y)dT 


(s)ds,  s = T/Y, 


Yx^(t)  t e [t^,  t^] 


(3.58) 


In  particular,  we  have 


2L2<to)  = X.2  = Yiii  = Yx^Ct-)  . 


(3.59) 


Step  ( ii) . 

From  (3.58)  we  see  that  if  any  component  of  x^  is  zero  on 
[t^,  ' then  the  corresponding  component  of  x,2  zero  on 

[t^,  tp^j^)  . Likewise,  if  any  component  of  is  greater  than  zero  on 
[tp,  tp^j^)  , then  the  corresponding  component  of  is  greater  than  zero 


on 


t 

P 


M 


t” 

P+1 


) . 


Step  (iii)  . 

By  virtue  of  the  fact  that  B(2C2^  " B(Xj^),  it  is  easily  verified 
that  dj]^2  defined  in  (3.57)  satisfies  the  complementary  slackness 
condition  (3.10).  Since  Is  defined  to  satisfy  a differential  equa- 
tion of  the  required  form,  and  dQ^^  satisfies  complementary  slackness, 
then  , T c [tj.,  tg]  , is  a legitimate  costate  trajectory. 


k 
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Step  (iv) 


The  relationship  between  and  backward  in  time  is 

ryt  ryt 

X2(yt)  = J a dT  + J dn^CT)  + 


by  (3.55)  and  (3.56) 


ryt  ryt 

I a dT  + y I dn^j^(T/y)  + yv^ 

•'t-  -'t. 


by  (3.5/) 


ft  ft 

y I a ds  + y I dn 


(s)  + yv^ 


where  s = T/y 


yX^(t)  V-  t e ftj.,  tj^I  . 


Definitions  (3.51)  imply 


,T  , 


(3.60) 


u = ARC  MIN  (X,  (T)  B u) 

"P  ucU 


T E (t\  P e [Of  If  •••f  f-1] 


= ARC  MIN  (yX  (t)B  u) 
ueU  ^ 


since  y positive  scalar 


= ARC  MIN  (^  (yr)B  u) . 
ueU 


(3.61) 


The  above  is  equivalent  to 
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i 

!, 

il 

^ = ARG  MIN  UTcDB  u)  (3.62) 

u£U 

T e [t^,  , p e [0,  1,  f-1] . 

Therefore,  our  supposition  regarding  the  optimality  of  U(x^^)  on  T(x2) 
is  verified.  Hence,  the  application  of  U(Xj^)  on  T(x^)  = {t^,  t^,  ..., 
tj}  forward  in  time  brings  the  state  from  ii2(tQ)  = to  “ £• 

■ Lemma  3. 1 

Lemma  3.2.  Let  the  inputs  £ = constant.  Suppose  for  some  x^^, 

— 2 ^ Xj^  £ 0,  £2  £ 0,  that 

U(x^)  = U(X2)  = U 

and 

B(Xj^)  = B(2i2^ 

If  X3  = yXi  t (l~Y)2i2'  ° lY  1 

then 

UCx^)  = U 

and 

BiXj)  = 8- 

Proof  of  Lemma  3.2; 

As  in  the  proof  of  Lemma  3.1,  we  assume  that  time  propagates  back- 
ward from  the  final  time.  Let  the  optimal  solution  which  brings  the 
state  from  x^^  to  0 be  characterized  by  the  control  sequence  and  switch 


time  set 
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U(x^)  = U = {^,  u^, 

T(Xj^)  = {t^,  ...,  tp 


(3.63) 


respectively  and  similarly  for  X2* 


UCx^)  = U = ^Hq'  -1'  •••' 


TCx^)  = ^t^,  t" t^}. 


(3,64) 


Associated  with  each  of  the  above  optimal  solutions  are  costate 
vectors  and  ^2  which  evolve  backward  in  time  according  to 


d^j^(T)  = ctdT  + dn^^(T) 


(3.65) 


T e [t^,  t^l 


(3.66) 


T e [t^,  t^l. 

The  proof  proceeds  along  similar  lines  to  the  proof  of  the  pre- 
vious lemma.  We  shall  produce  a switching  time  set  and  a costate 

history  ^ ^ ^ for  which  U is  an  optimal  control  sequence  which  brings 

X2  = + ^^“Y)2L2  — — ■ These  sets  are 


'(X3)  = {t-  , t- f''  } 


(3.67) 


where  t'"  = vt'  + (l-v)t" 

P ' P P 


(3.68) 


P e [0,  1 fl 


(3.69) 


where  * (1-Y)^2**'f^ 
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and 


At' 

dn,(t"'  + T)  = Y dn,  (f  + T-^  T) 

— j p —1  p tit 

P 

At" 

+ (1-Y)  ^ T) 

p 


T e [0,  At^'  ) , P e [0,  1,  . , f) 


where  we  denote 


Notation  3.2 


At'  = t'  - f , 

P P P-1 


At"  = t"  - t"  , 

P P P-1 


V-  p e [0,1, . . . , f ] 


At"'  = t"'  - t"'  , 

P P p-1 


From  equations  (3.72)  and  (3.68)  we  obtain 


At"'  = YAt’  + (l-Y)At". 
p ' p ' P 


(3.71) 


(3.72) 


(3.73) 


To  show  that  indeed  the  above  set  optimally  brings  to  0^  we  have 
to  show  the  following  assertions: 


(i)  The  application  of  U on  T(x^)  drives  the  state  from  0 

to  5c^  = + (1^)X2  backward  in  time. 

(ii)  ^(^3)  = S- 

(iii)  and  satisfy  all  of  the  nececessary  conditions. 

(iv)  U is  optimal  on  T(^j)- 


step  (i) . 

Since  our  system  is  time  invariant,  we  set  t^  = t^  = t^'  = t^  as 

the  initial  time  in  the  backward  sense.  We  now  denote  by  , ^2^’^ 

and  the  state  trajectories  resulting  from  the  application  of  U 

on  T(jCj^),  T(x^2^  respectively  and  designate  ^ + a. 

Integration  of  the  state  dynamics  backward  in  time  yields 


x,(f"  + t) 
-3  p 


- 52  X At'"  - X , t 
q=f-l-^-^  q -^>-1 


Zx  , (YAt'+(l-Y) At") -X  ,t 
,-q-l  q q -p-1 


q=f-l 


by  substituting  (3.73) 


= Yx,  (f)  + a-Y)x_(t")  - X 

—Ip  “2  p -p-1 

t e (0,  At’"  ) . 

P 


Also  from  (3.73)  we  have 


+ (1-Y)  = 1 


so  that  the  above  expression  now  becomes 


x,(t’"  + t) 

-3  p 


= yx  (t' ) + (1-y) X- (t") 
'-1  p -2  p 

At'  At" 

P P 


At'  At" 

= Yx,  (t*  + T-^  t)  + (l-Y)x^(t"  + 7-^  t) 
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t e (0,  At'"  ) . (3.74) 

P 

In  particular,  we  obtain  the  desired  value  at  t^'  : 

=213=  "^-1  ^ ^ 

Step  (ii) . 

Since  S(Xj^)  = BCx^)  = 8,  if  any  component  of  x^^  is  zero  on 
[t' , t'  ,),  then  the  corresponding  component  of  x.  is  zero  on  [t",t”,,), 

pp+1  _2  p p^.J^ 

and  vice  versa;  also,  from  (3.74)  we  see  that  under  this  circumstance 
the  corresponding  component  of  is  zero  on  [t^'  , • Likewise  for 

components  strictly  greater  than  zero.  Hence,  ^(x^)  = 8. 


Step  (iii). 

By  virtue  of  the  fact  that  8(x^)  = 8 it  is  easily  verified  that  dn_2 
as  defined  in  (3.71)  satisfies  the  complementary  slackness  condition 
(3.10).  Since  is  defined  to  satisfy  a differential  equation  of  the 
required  form  and  d^^  satisfies  complementary  slackness,  then  (t ) , 

T e ] f is  a legitimate  costate  trajectory. 


Step  (iv). 

Integrating  the  costate  eijuation  (3.69)  backward  in  time  on 

t e [0,  At"’  ) we  obtain 
P 


X,(t"’  + t)  = ) + / n dr  + / dr, 

-3  p -3  p /t"' - /t"'  - 

! )' 

r r 

I a dT  +>  I dn^  (t 


(t  + T) 


= A,(t"’  ) + 
-3  p 


At  ■ 
±- 
>.  " ’ 

■p 


l2 

P 


Now  perforin  the  change  of  variables 


" At'" 


®2  At'"  ^ 


and  note  that  T = ys-,  + (l-Y)s„  by  (3.73).  Then 


£=t 

01  d(ys 

At'  P 

+ Y J dai^^^p  + J 

J.  r*  •'4.1M 


+ (1-Y)s2) 


S ='  * - ti 

2 At'" 


dn,(t"  + s„) 
-2  p 2 


S ^ * t 

At'" 


(a  ds,  + dn,  (f  + s ) ) 

— 1 ~1  P i 


+ (1-  Y) 


, 

®2  At'"  ^ 
P 


(a  ds  + dn_(t"  + S-) ) 
— Z ^Z  p z 


t C [0,  At"'  1 . 

p 


(3.76) 


From  (3.65)  and  (3.66)  we  obtain  by  backward  integration 

r % 

/■'"l  At'"  ^ 

At’  1 P 


p t' 


(a  ds,  + dn,  (f  + s ) ) 
— 1 p 1 


(3.77) 
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and 


At" 

I ^ <2 

Jt'" 

^ P 


(a  ds^  + dn_(t"  + s^: 


- 2 


•2  P 


T e [0,  At'"  ) . 

P 

Substituting  (3.77)  and  (3.78)  into  (3.76)  obtain 

At’ 

is";'  « ■ is";  I • »ii";  * sj|- « - 

At" 

+ d-Y)A2<t"  + ^ t)  - d-Y)A2(t") 
^ P 


2 

(3 


T e [0,  At'"  ) . 

P 


(3, 


Upon  substitution  of  the  end  condition  (3.70)  at  t^  = t^  = t^  = t^' 
the  costate  propagates  in  the  following  fashion  backward  in  time: 

Aa(t-  t t)  = YA^(t'  + ^ t)  t d-Y)A2(t^  + ^ t)  (3, 


V-  P e (0,  1,  .. . , f 1 , t e [0,  At"'  ) . 


We  now  verify  that  for  A3  given  by  the  differential  equation  i 
with  end  condition  (3.70),  U is  an  optimal  control  sequence  on  T(j^. 
By  the  definitions  of  U,  T(Xj^)  and  T(2i2^  have 


U*  = ARC  MIN  A, (t'  + t)B  u,  t E [0,  At') 
“P  udi  P , P 


(3. 


and 


u*  = ARC  MIN  A.(t"  + t)B  u,  t e [0,  At"). 


ucU 


-2  p 


»j. 

.78) 


79) 

t 

80) 

3.69) 

). 

81) 


(3.82) 
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I 

I 


,1 


! 

i 

I 


Equations  (3.81)  and  (3.82)  imply 

YA^(t'  + t)B  u*  < YAT(t'  + t)B  u 
—Ip  p — — Ip  E 


(3.83) 


and 


V u e U,  t e [0,  At' ) 
-P  P 


(l-Y)A^(t"  + t)B  u*  < d-Y)A'!(t"  + t)B  u 

— 2p  p—  — 2p  p 

V-  u e U,  t e [0,  At") . 

“P  P 


Equations  (3.80),  (3.83)  and  (3.84)  give 


(3.84) 


A'?(t"'  + t)B  u*  < A^(t"‘  + t)B  u 
-3p  p 3p  p 

V-  u c U,  t e [0,.  At'"  ) , 

-P  P 


(3.85) 


that  is 


u*  = ARG  MIN  A::(t'"  + t)  , t e [0,  At'"  ) 

“P  ueU  -3  P P 


(3.86) 


V-  p £ [0,  1,  . ..  , f] 


Therefore,  with  defined,  U is  an  optimal  control  sequence 

which  when  applied  forward  in  time  on  T(j{2)  = ,t^"  ,...,  t^  } brings 

the  state  from  ^ ~ — 3 — 3**"f'  ^ ' ■ Lemma  3.2 

Lemma  3.1  implies  that  the  regions  B)  are  cones  and  Lemma  3.2 

implies  that  these  cones  are  convex.  As  each  region  is  convex,  the 
interface  between  any  two  adjacent  regions  must  be  a portion  of  a hyper- 
plane through  the  origin  which  belongs  to  one  region  or  the  other  (we 


1 

i 

] 

] 


assume  for  technical  reasons  that  the  number  of  regions  is  finite) . 


Therefore,  we  have  established  the  convex  polyhedral  conical  nature  of 


the  regions. 


Theorem  3.4 


In  preparation  for  development  of  the  feedback  technique,  we  pre- 
sent the  following  corollary  to  Theorem  3.1  which  narrows  down  the 
freedom  of  the  costates  at  the  final  time. 


Corollary  3.2  If  any  state,  say  x^,  is  strictly  positive  on  the  final 

k 

time  interval  t^)  of  an  optimal  trajectory,  then  A^(t^)  = 0. 


Proof : 


Consider  a specific  state  x^  satisfying  the  hypothesis.  Then 
since  x^(t),  t e ^f^  ' constant  by  Corollary  3.1,  we  must 


Xi(tf)  < 0. 


figure  3.6  Typical  State  Variable  Approaching 
Boundary  at  t 


Then  there  must  exist  a directed  chain  of  links  from  node  i to 
node  k (arbitrarily  denote  them  by  {(i,  i+1),  (i+1,  i+2) , ...,  (k-l,k)}) 
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carrying  some  messages  with  destination  k,  that  is 

k 


In  order  for  the  above  control  to  be  optimal,  we  must  have  (by  (3.41)- 
(3.43)) 

^i+1 ° 


■\-i<V  10- 


From  the  above  we  conclude 

We  now  proceed  to  show  by  contradiction  that  = 0.  Suppose 

k 

^i(tf)  > 0.  Then  the  transversality  condition  ^ X^(t^)x?(t^)  = 0 

i/j  ^ ^ 

• 1 • ’ll  P 

implies  that  there  must  be  at  least  one  x^(t^)  < 0 such  that  X (t^)  < 0. 

s. 

But  the  above  reasoning  applied  to  implies  that  ^^(^^)  ^ 0.  Hence, 


a contradiction. 


Corollary  3.2 


3.3  Backward  Construction  of  the  Feedback  Space  for  Constant  Inputs 
3.3.1  Introductory'  Examples  of  Backward  Boundary  Sequence 
Technique 

The  construction  of  regions  of  the  type  Y'(U,  6)  will  play  an  impor- 


tant role  in  the  synthesis  of  the  feedback  space. 


A basic  observation 
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with  regard  to  these  regions  is  that  they  are  functions  of  the  entire 
future  sequence  of  controls  which  carry  any  member  state  optimally  to 
zero.  This  general  dependence  of  the  current  policy  upon  the  future 
is  the  basic  dilemma  in  computing  optimal  controls.  This  problem  is 
often  accommodated  by  the  application  of  the  principle  of  dyncimic  pro- 
gramming, which  seeks  to  determine  the  optimal  control  as  a function 
of  the  state  by  working  backward  from  the  final  time.  The  algorithm  to 
be  developed  employs  the  spirit  of  dynamic  programming  to  enable  con- 
struction of  regions  of  the  type  'tflU,  B)  from  the  appropriate  set  of 
optimal  trajectories  run  backward  in  time.  These  trajectories  are 
fashioned  to  satisfy  the  necessary  and  sufficient  conditions  of 
Theorem  3.1,  as  well  as  the  costate  boundary  condition  at  t^  given  in 
Corollary  3.2. 

We  motivate  the  backward  construction  technique  with  several  two 
dimensional  examples  which  introduce  the  basic  principles  involved. 

In  fact,  throughout  the  thesis  important  concepts  will  be  illustrated 
with  the  simplest  possible  examples  whenever  the  interests  of  clarity 
can  be  served.  The  reader  is  encouraged  to  study  these  examples  care- 
fully, as  they  comprise  an  essential  part  of  the  exposition.  Also, 
one  is  able  to  grasp  the  general  geometrical  notions  more  readily  with 
a simple  two  or  three  dimensional  picture  in  mind. 


Example  3.1 


FiijUre  3,7  riii.plrj  ,Sinql<  .)P!3tindtio..  :.etwor)i 


The  network  as  pictured  in  Figure  3.7  has  a single  destination, 
node  3;  hence,  we  can  omit  the  destination  superscript  "3"  from  the 
state  and  control  variables  without  confusion.  For  simplicity,  we 
assume  that  the  inputs  to  the  network  are  zero,  so  that  the  dynamics  are 


Xi(t)  = -u^3(t)  - U32(t)  + U23(t) 


x^Ct)  = -U23(t)  + u^,(t)  - u.,,(t) 


(3.87) 


1 
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with  control  constraints  as  indicated  in  Figure  3.7, 
is  defined  as 


D 


{xj^(t)  + X2(t)}dt. 


Let  the  vector  notation  be 


The  cost  function 


(3.88) 


We  wish  to  find  the  optimal  control  which  drives  any  state 
xlt^)  ^ 0 to  x(t^)  = ^ while  minimizing  D. 

As  our  intent  is  to  work  backward  from  the  final  time,  we  list  all 
possible  situations  vrhich  may  occur  over  the  final  time  interval 

[t^  t^]  with  respect  to  the  states  and  x^.  Recall  from  Corollary 

• • 

3.1  that  the  state  rates  x^  and  x^  are  constant  on  all  intervals. 


(i) 


x^  < 0 


x^  = x^  = 0 


(ii)  x^  < 0 


x^  = x^  = 0 


(iii)  x^  < 0 


x^  < 0. 


These  are  certainly  the  only  situations  which  may  occur  on  the 


-95- 


final  time  interval,  although  we  have  no  way  of  knowing  a priori  which 
are  consistent  with  the  optimality  conditions.  We  do  know  by  virtue  of 
sufficiency  that  any  trajectory  which  is  optimal  has  associated  with 
it  costates  which  satisfy  the  necessary  conditions;  and  that  any  tra- 
jectory which  satisfies  the  necessary  conditions  is  indeed  optimal. 
Hence,  our  first  task  in  the  analysis  of  each  of  these  cases  shall  be 
to  attempt  to  produce  a set  of  costates  for  which  the  control  which 
solves  the  appropriate  form  of  (3.37)  gives  rise  to  the  specified  tra- 
jectory on  some  non-zero  interval  of  time  preceding  the  final  time. 

We  now  consider  these  situations  one  at  a time: 

(i)  Xj,(T)  < 0,  X2(t)  = x^Ct)  =0,  T c [t^_j^,  t^J. 

This  situation  is  depicted  as  part  of  Figure  3.8  . We  begin  by 
considering  the  time  period  ir'  ^ general  sense  without 

actually  fixing  the  switching  time  This  is  simply  the  time  period 

corresponding  to  the  final  bang-bang  optimal  control  which  brings  the 

• • 

state  to  zero  with  Xj^  < 0 and  x^  = X2  = 0.  We  now  set  out  to  find  if 
there  is  a costate  satisfying  the  necessary  conditions  for  which  this 
situation  is  optimal;  and  if  so,  to  find  the  value  of  the  optimal  con- 
trol. The  linear  program  to  be  solved  on  T e [t^  , t^l  is 

U*(T)  = ARG  MIN  [X.(t)x, (T)  + A-(t)x.(t)) 
ucU  ^ ^ 2 2 

= ARG  MIN  [(A^d)-!  (T))Uj^2<^>  + (A^(T)-A  (T))u2j^(T) 
ucU 


(3.89) 


te  Trajectory  Pair 
3.1,  Case  (i) 
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I 


i 


Now,  the  stipulation  < 0 tells  us  from  Corollary  3.2  that 

= 0 (3.90) 

and  since  is  bn  an  interior  arc,  equation  (3.32)  gives 

X^(T)  = -1  TO  . (3.91) 

This  is  shown  in  Figure  3.8  . Now,  since  we  specify  x^  = 0 on  this 
interval,  iis  bostate  equation  is 

-dX^d)  = 1 dT  + dri2(T)  (3.92) 

dn2(T)  < 0 

X2(t^)  = V2  free  T e 

where  H2  is  a possibly  discontinuous  function.  We  now  submit  that  the 

costate  value  X^d)  = ^^(T)  = 0,  T £ i-®  such  that  there 

exists  a solution  to  (3.89)  for  which  x^^  < 0 and  x^  = 0.  The  reader 
in  fact  may  verify  that  X^(T)  = 0,  T e [t^  t^] , is  the  only  possible 

value  since  X2(T)  > 0 or  X2(t)  < 0 imply  by  (3.89)  and  (3.87)  that 

x^  < 0 and  > 0 respectively.  Before  presenting  the  specific  solution 
we  verify  that  X2(t)  = X2(t)  = 0,  t e ® costate  value 

which  is  consistent  with  equation  (3.92).  First,  the  required  final 
condition  X2^*^f)  = 0 is  acceptable  since  the  necessary  conditions  leave 
it  entirely  free.  Also,  the  differential  equation  X2(t)  = 0 may  be 
realized  with  dri2(T)  = -dt,  which  is  consistent  with  (3.92).  This 
costate  trajectory  is  pictured  as  part  of  Figure  3.8  . Now,  with  the 


i 


\ 


\ 


\ 


costates  so  determined,  one  solution  to  (3.89)  is 
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ud) = (0.5,  0.0,  1.0,  0.5) 


Xj^d)  = -1.5  X2d)  = 0 
T e t^] . 


(3.93) 

(3.94) 


We  emphasize  that  the  above  solution  is  only  one  among  an  infinite  set 
of  solutions  to  (3.89).  However,  it  is  the  solution  which  we  are 
seeking.  We  now  make  an  important  observation  regarding  this  solution. 
Since  Xj^(T)  = -1  and  ^^(T)  = 0 for  T £ *"f^  ' control  (3.93) 

remains  optimal  on  T e (-“,  t^]  . But  as  t^  ^ ^)  -*•  °°. 

Thinking  now  in  forward  time,  this  implies  that  any  initial  condition 
on  the  x^-axis  can  be  brought  to  zero  optimally  with  the  control  spe- 
cified in  (3.94).  We  have  therefore  determined  the  optimal  feedback 
control  for  all  points  on  the  xj^-axisl  This  is  indicated  in  Figure  3.9. 

Suppose  now  that  we  wish  to  consider  a more  general  class  of  tra- 
jectories associated  with  the  end  condition  under  discussion.  What  we 
may  do  is  to  temporarily  fix  t^  ^ and  stipulate  that  the  control  on 
[t^_2/  t^_^)  has  x^  negative;  that  is,  insist  that  x^  "leave  the 
boundary"  bac)cward  in  time.  As  before,  the  initial  time  t^  2 
segment  [t^  2'  left  free.  The  program  to  be  solved  is  (3.89) 

with  T e [t^  2'  2^)  • Now,  since  x^^  is  on  an  interior  arc  across 

t^  by  (3.32)  its  costate  must  be  continuous  across  t^  , that  is 


Xj^(tf-i)  X^(t^_j^)  - t^  t^_^. 


(3.95) 


Since  (3.92)  allows  for  only  positive  jumps  of  X2  forward  in  time,  we 
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have 

" °*  (3,96) 

Also,  since  both  and  are  on  interior  arcs  on  [t^  2' 
tion  (3.32)  gives 

Xj^(T)  = -1 

T e [t  , t ).  (3.97) 

X2(t)  = -1 

The  resultant  costate  trajectory  is  depicted  in  Figure  3.8.  We  now 
perform  the  minimization  (3.8'^)  ' r T e [t^  2»  • Since 

Xj^(t)  > X2(t)  > 0»  T E ^^f-2'  *■!  solution  to  (3.89)  is 

u(t)  = (0.5,  0.0,  1.0,  1.0)'^  (3.98) 

so  that 

Xj^(t)  = -1.5  ^2^^^  ~ -0,5.  (3,99) 

Therefore,  the  optimal  control  gives  X2(t)  < 0,  which  is  the 
situation  which  we  desire.  Once  again,  we  see  that  the  control  is 
optimal  for  t e (-°°,  . Since  “ 3.0,  upon  leaving  the  x^^ 

axis  bac)tward  in  time  the  state  travels  parallel  to  the  line  x^^  - 3x^  = 
0 forever.  Now,  recall  that  t^_^  is  essentially  free.  Therefore,  from 
anywhere  on  the  x^  axis  the  state  leaves  parallel  to  x^^  - 3x2  ^ ° with 
underlying  optimal  control  (3.98),  Thiniting  now  in  forward  time,  this 
implies  that  any  initial  condition  lying  in  the  region  between  the 
line  Xj^  - 3X2  = 0 and  the  Xj^-axis  (not  including  the  x^^-axis)  may  be 
brought  optimally  to  the  x^^-axis  with  the  control  (3.98).  See 


i 
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Figure  3.9.  Once  the  state  reaches  the  x^^-axis,  the  optimal  control 
which  subsequently  takes  the  state  to  zero  is  given  by  (3.93) . There- 
fore, we  have  now  managed  to  fill  up  the  space  between  the  line 


so  that 


Xj^(T)  = -1.0  - 

x^Ct)  = -1.0  + ■ ^21^^^' 


(3.104) 


Hence,  we  have  encountered  the  non-uniqueness  in  the  optimal  con- 
trol discussed  in  a general  sense  at  the  end  of  Section  3.2.1.  The 
values  of  u^^  ^i2l  completely  arbitrary  as  long  as  they  remain 
within  their  constraints.  Moreover,  for  any  T e (-“,  t^]  the  entire 
set  of  optimal  controls  (3.103)  remains  optimal.  In  this  case,  the 
optimal  directions  by  which  the  state  leaves  the  origin  lie  between 
x^/x^  = 3.0  and  x^/x^  = 3.0,  that  is,  between  the  lines  x^  - 3x^  = 0 
and  x^  - 3Xj^  = 0.  As  before,  we  now  switch  our  consciousness  to  for- 
ward time  and  realize  that  for  any  point  lying  between  the  lines 
x^  - Ix^  = 0 and  x^  - 3x^^  = 0 (not  including  these  lines)  the  complete 
set  of  controls  (3.103)  is  optimal.  For  any  point  on  the  line 
x^  - 3x^  = 0 only  the  control  which  has  Uj^2  = 0*5  and  u^^^  = 0.0  is 
opti  3l.  Li)cewise,  for  any  point  on  x^  ~ 3x^  = 0 only  the  control 
which  has  u^^2  = *3.0  and  U2j^  = 0.5  is  optimal.  See  Figure  3.9. 

Having  filled  up  the  entire  space  with  optimal  controls  region 
by  region,  the  specification  of  the  feedbaclc  control  is  now  complete. 

• Example  3 . 1 


Example  3 . 2 

The  networlc  is  the  same  as  for  Example  3.1,  but  the  cost  func- 


tional is  taken  as 
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I 


D 


{2x^(t)  + x^Ct) }dt. 


(3.105) 


As  in  Example  3.1,  we  take  the  approach  of  working  backward  from  the 
final  time,  beginning  with  the  three  possible  situations  which  may 
occur  at  that  time. 


(i)  x^(T)  < 0,  x^Ct)  = = 0,  T e t^] 

The  linear  program  to  be  solved  over  the  final  time  interval 
T e (3.89)  with  X^(r)  and  X2(t)  appropriately  determined. 

The  final  condition  (3.90)  applies,  but  since  the  weighting  on  x^  is 
= 2,  the  appropriate  differential  equation  for  X^  is 

Xj^(T)  = -2  T C [t^_^,  t^)  . (3.106) 

Now,  X^(T)  is  determined  in  the  same  fashion  as  X^(j)  in  case  (i)  of 
Example  3.1.  That  is,  the  value 

X^(T)  =0  T e t^]  (3.107) 

allows  the  solution  to  (3.89)  to  be  such  that  x^(T)  = 0,  T e • 

Consequently,  the  optimal  control  (3.93)  applies  here.  The  feedback 
control  on  the  x^^-axis  is  therefore  assigned  in  the  same  fashion  as 
in  Example  3.1.  See  Figure  3.12. 

Let  us  now  allow  x^  to  leave  the  boundary  backward  in  time  at 
t^  j.  This  situation  (as  well  as  the  previous  one)  is  illustrated  in 
Figure  3.11.  The  linear  program  to  be  solved  is  as  always  (3.89).  In 


this  case  we  have 


u = (0,0. 5, 1.0, 1,0) 
= -0.5 


Feedla.-is  .'-olution  for  I'xamplt  ^.2 
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'I'Vi'  • \'Vi’  ■ "‘'f  - 'f-i' 


(3.108) 


Since  both  and  x^  are  on  interior  arcs  over  this  interval,  their  dif- 
ferential equations  are 


i 


Aj^(T)  = -2 

A2(t)  = -1 


T e [t 


f-2' 


^f_l> • 


(3.109) 


Also,  as  before,  all  that  matters  in  the  solution  of  the  linear  prograun 

is  that  A^{t)  > > 0,  T e [t^  2'  ^f-i^ ' Therefore,  the  solution  is 

given  by  (3.98)  and  this  solution  applies  on  the  conical  region 
*1 

{x| 0 < X2  5.  3“^  argued  in  Example  3.1.  See  Figure  3.12. 


(ii)  x^d)  < 0,  x^(t)  = x^(T)  = 0,  T e • 

The  details  of  this  situation  are  depicted  in  Figure  3.13. 
We  )cnow  from  Corollary  3.2  that 


W ■ 


(3.110) 


cuid  from  (3,32)  that 

A^(T)  = -1  T £ t^].  (3,111) 

We  now  may  find  by  the  process  of  elimination  that  the  only  value  of 
Aj^(t),  t e which  x^^  = 0 is  optimal  is: 

X^(t)  = Aj^(t)  =0  T £ t^l  . (3.112) 

Therefore,  the  solution  to  (3.89)  is  the  same  as  in  Example  3.1, 


case  (ii)  and  the  control  on  the  x^-axis  is  assigned  in  identical 
fashion.  See  Figure  3.12. 


As  the  next  step,  we  now  stipulate  that  Xj^  leaves  the  boundary 
backward  in  time  at  Since  ^ have  from  (3.32)  that 


^2^^f-l^  " = t,  - t 


2 f-1' 


■f-l‘ 


(3.113) 


Since  costate  jumps  can  only  be  positive  in  forward  time,  we  must  have 

>^l(t“_i)  = 0.  (3.114) 

Also,  since  x^^(T)  > 0,  x^(T)  >0,  T e [t^  2'  ' 

Aj^(t)  = -2  I 

I T C • (3.115) 

X2(t)  = -1  I 

See  Figure  3.13.  We  now  notice  a fundamental  difference  between 
this  and  the  previous  situations.  At  some  time  before  t^_^  the  sign  of 
- X2<t))  changes,  which  implies  that  the  solution  to  the  linear 
program  (3,89)  changes  at  that  time.  Therefore,  t^  ^ allowed  to 

run  to  -00,  but  is  actually  the  time  at  which  the  costates  cross  and  the 
control  switches.  The  optimal  controls  and  state  velocities  on  either 
side  of  the  switch  are: 


T e [V2'  Vl>  = 


u = (0.0,  0.5,  1.0,  1.0) 


X = -0.5 


X,  = -1.5 


(3.116) 

(3.117) 


Finally,  (3. 122) - ( 3 . 124 ) give 
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~ " °*  (3.125) 

That  is,  the  switch  of  control  corresponding  to  the  time  t^  ^ always 
occurs  when  the  state  reaches  the  line  (3.125).  Therefore,  backward 
in  time  the  state  leaves  from  anywhere  on  the  axis  with  control 
(3.116)  and  associated  rate  (3.117).  The  direction  of  travel  is 
actually  parallel  to  the  line  - 3x^  = 0.  Upon  reaching  the  line 
x^  - 6Xj^  = 0,  '^.e  control  switches  to  (3.118)  and  the  state  travels 

parallel  to  the  line  x^^  - Ix^  = 0 forever.  This  sequence  is  illustrated 
for  a sampled  trajectory  whose  portions  are  labeled  in 

Figure  3.12  and  Figure  3.13. 

From  these  observations,  the  following  may  be  inferred  about  the 
feedback  solution:  The  control  (3.116)  is  optimal  anywhere  within  the 

region  bounded  by  the  x^-axis  and  the  line  ~ = 0,  not  including 

the  x^-axis.  The  control  (3.118)  is  optimal  euiywhere  within  the  region 
bounded  by  the  lines  x^  ~ 6x^  = 0 and  x^^  - 3x^  = 0 not  including  the 
former  line.  These  regions  are  indicated  in  Figure  3.12. 

Since  the  entire  space  has  now  been  filled  up  with  optimal  con- 
trols, the  feedback  specification  is  now  complete.  The  reader  may 
verify  by  a simple  exercise  that  case  (iii)  merely  provides  the  optimal 
control  on  the  line  x^^  - Ix^  = 0,  which  we  already  have. 

• Example  3 . 2 

We  now  summarize  the  content  of  the  preceding  examples.  By 


starting  at  the  final  time,  we  have  allowed  states  to  leave  the  boundary 
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backward  in  time  and  have  computed  the  corresponding  optimal  trajec- 
tories as  time  runs  from  zero  to  minus  infinity.  In  the  instances 
when  the  control  did  not  switch  over  this  interval,  we  were  able  to 
construct  one  conical  region  of  common  optimal  control  (as  well  as 
conunon  control  and  boundary  sequences  which  take  any  member  state  to 
zero).  When  the  control  did  switch,  as  in  case  (ii)  of  Example  3.2, 
two  such  regions  were  constructed.  By  considering  enough  cases,  we 
were  able  to  fill  up  the  entire  space  with  regions  of  common  optimal 
control,  thereby  providing  the  feedback  solutions.  The  primary  diffi- 
culty in  calculating  the  optimal  controls  by  the  necessary  conditions 
was  in  the  determination  of  the  proper  value  of  the  costates  correspon- 
ding to  states  specified  to  be  on  boundary  arcs. 

3,3.2  Technique  for  the  Backward  Construction  of  the  Feedback 
Space 

The  examples  of  the  previous  section  suggest  an  approach  by  whicli 
the  feedback  solution  to  (3.1)- (3.6)  may  be  synthesized.  In  this  sec- 
tion, we  take  our  lead  from  these  examples  to  suggest  a general  method 
for  the  synthesis  of  a feedback  solution.  We  then  present  the  basic 
structure  of  an  algorithm  which  may  realize  the  method.  Several  ques- 
tions are  identified  which  must  be  answered  in  order  to  implement  the 
algorithm. 

3. 3. 2.1  Prel iminaries 


We  begin  with  a definition  which  describes  a class  of  subsets  of 
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which  shall  comprise  the  building  blocks  of  the  feedback  space: 

Definition  3.6:  A set  C R*^,  is  said  to  be  a feedback  control 

region  with  control  set  51  C U,  if  the  following  properties  hold: 

(i)  Consider  any  two  points  3t ^ Int  . Suppose  U(Xj^)  = U 
with  associated  switch  time  set  T(}c^)  . Then  UCx^)  = U for 
some  switch  time  set  Tlx^)-  Here  U(x)  and  T(x^)  are  as  defined 
in  (3.46) . 

(ii)  = ^<2.2^ 

vrtiere  6(x)  is  defined  in  (3.48). 

(iii)  Any  control  u e 51  that  keeps  the  state  inside  the  region  for 
a non-zero  interval  of  time  is  an  optimal  control.  Formally,  if 
x(t)  e .‘/I  then  any  e 51  is  an  optimal  control  if  there  is  an 

E > 0 such  that  under  the  application  of  u , ^(t+e)  e .‘fi. 

The  reader  may  verify  that  all  of  the  regions  constructed  in 
Excunples  3.1  and  3.2  are  feedback  control  regions  in  the  sense  of 
Definition  3.6.  Our  goal  now  is  to  specify  an  algorithm  which  will 
construct  feedback  control  regions  for  the  general  problem.  As  a 
corollary  to  Theorem  3.4  we  obtain  quite  readily  the  following  geometri- 
cal characterization  of  feedback  control  regions. 

Corollary  3.3  Let  the  inputs  ^ = constant.  Then  the  feedback  control 
regions  of  Definition  3.6  are  convex  polyhedral  cones  in 


Proof : From  properties  (i)  and  (ii)  of  Definition  3.6  we  see  that  any 
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1 


feedback  control  region  is  the  intersection  of  a set  of  regions  of 
the  type  8),  From  Theorem  3.4  we  know  that  all  regions  of  this 

type  are  convex  polyhedral  cones  in  the  state  space;  therefore,  the 
non-empty  intersection  of  any  set  must  also  be  a convex  polyhedral 
cone.  ■ Corollary  3.3 

Definition  3.7:  A segment  of  an  optimal  trajectory  is  that  portion 

which  occurs  on  the  time  interval  between  two  successive  switch  times 
T and  T , not  including  the  time  instant  T , . 

p p+1  p+1 

The  basic  concept  underlying  the  construction  of  the  feedback 
solution  is  now  presented: 

Constructive  Dynamic  Programming  Concept 

Construct  a set  of  backward  optimal  trajectories,  each 
starting  at  the  final  time  t^,  among  which  all  possible 
sequences  of  states  leaving  the  boundary  backward  in  time, 
both  singly  and  in  combination,  are  represented.  Each  seg- 
ment of  every  optimal  trajectory  is  utilized  to  construct  a 
feedback  control  region  with  associated  optimal  control  set. 
These  feedback  regions  are  convex  polyhedral  cones,  and  the 
union  of  all  such  cones  is  the  entire  space. 

3 . 3 . 2 . 2 Description  of  the  AlgorjUhm 

The  algorithm  by  which  the  above  method  may  be  realized  is  now 


presented.  We  emphasize  that  several  of  the  procedures  which  the 
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algorithm  calls  for  are  not  included  in  this  presentation,  but  shall 
be  singled  out  for  future  consideration  as  the  discussion  proceeds. 

For  clarity,  the  discussion  in  this  section  is  divided  into  three  parts, 
which  we  now  list: 

(i)  General  description  of  a step  of  the  algorithm. 

(ii)  Detailed  description  of  a step  of  the  algorithm. 

(iii)  Rule  governing  the  frequency  of  execution  of  the  step,  and 
proof  that  the  algorithm  realizes  the  constructive  dynamic 
programming  concept. 

(i)  General  description  of  a step  of  the  algorithm 

Each  step  consists  of  two  basic  operations.  They  are: 

• Operation  1 

Allow  a certain  set  of  state  variables  to  leave  the 
boundary  ^ = 0 bac’vwaru  in  time  and  determine  the  subsequent 
optimal  trajectory  (trajectories)  backward  in  time  assuming 
that  no  other  state  variables  leave  the  boundary  as  time  runs 
to  minus  infinity. 

• Operation  2 

Utilize  the  relevant  information  associated  with  the  optimal 
trajectory  (trajectories)  of  Operation  1 to  construct  one  or 
more  feedback  control  regions  with  associated  control  sets  re- 


I 


I 


ferred  to  in  the  constructive  dynamic  programming  concept. 
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(ii)  Detailed  description  of  a step  of  the  algorithm 

We  first  establish  some  shorthand  notation  which  facilitates 
the  discussion. 


Notation  3.3 


Ip  = > 0,  T t [Tp, 


(3.126) 


Is  the  set  of  state  variables  which  travel  on  interior  arcs  during  the 
segment  corresponding  to  [t^,  • 


Notation  3.4 


L = {x^lx^  leaves  the  boundary  at  time  t } 
p i 1 P 


(3.127) 


the  set  of  state  variables  designated  to  leave  the  boundary  at  t 


With  B as  defined  in  (3,47)  we  ta)^e  note  of  the  following  rela— 
P 


tionships : 


Ip-1  ~ Ip^  ^P 


(3.128) 


L c B 
p p 


(3.129) 


6 , = S //.  . 

p-1  p p 


(3.130) 


Notation  3 . 5 


a = cardinality  of  I . 
P P 

p = cardinality  of  Lp. 


(3.131) 


(3.132) 
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Notation  3.6 


is  the  set  of  optimal  control  vectors: 


{u*|u*  = ARG  MIN  X'^(T)B  u(T),  T £ [T  , T j}.  (3.133) 

— — — n P+1 


ueU 


Notation  3.7 

is  u.e  feedback  region  determined  in  Operation  2 by  the  para- 
meters  of  the  trajectory  segment  on  [T  , T ,)  with  associated  control 

p p+1 

set  . 

P 

a 

From  notations  3.3  and  3.7  we  see  that  ,^/{  C.  R where  the  basis 

vectors  of  R are  the  elements  of  I . This  situation  is  illustrated 

P 

in  Figure  3.14.  Due  to  the  pictorial  limitation  of  three  dimensions, 

a 

n P ■ 

K xs  represented  in  two  dimensions  in  Figure  3.14.  The  remainder 

of  R is  R where  the  basis  vectors  of  R are  the  elements 

n-o 

of  8^.  The  third  orthogonal  axis  represents  R Basis  sets  are 

in  parentheses. 

In  order  to  describe  a single  step  (that  is  a sequence  of  Opera- 
tions 1 and  2)  we  pick  up  the  algorithm  at  a point  at  which  the  con- 
struction of  a feedback  control  region  has  just  been  completed  in 
the  previous  step.  We  further  stipulate  that  on  the  current  optimal 
trajectory  (trajectories)  the  state  variables  of  are  on  interior 
arcs  and  those  of  8^  are  on  the  boundary.  Refer  again  to  Figure  3.14. 


In  detail,  the  two  operations  are: 


i Operation  1 


Suppose  that  we  choose  a specific  set  of  state  variables 


to  leave  the  boundary  backward  in  time.  From  Figure  3.14  we 


visualize  that  the  statement  "allow  to  leave  the  boundary  backward 


in  time"  is  equivalent  to  the  state  variables  in  to  leave 


the  region  ./P  backward  in  time".  That  is,  the  state  vector  leaves 

P - - 

C ■t'P 

the  subset  • of  R ^ to  travel  into  R ^ P.  Looking  ahead,  in 
P 


Definition  3.8  we  shall  describe  a division  of  into  subregions 


with  respect  to  L^,  This  definition  is  deferred  until  after  the 


description  of  Operation  2 since  several  notions  developed  in  that 


operation  are  required.  The  question  of  determining  subregions  is 


confronted  in  Sections  3. 3. 2.2  and  3. 3. 3. 2. 


For  the  purpose  of  the  present  discussion,  let  us  assume  that  we 


have  divided  into  s subregions  with  respect  to  L . Denote  these 
P P 

12  s 

subregions  by  (L  ),,:>?  (L  )»  ...,<!<*  IL  ),  where  the  depend  snce  of 
p p p p P P 


the  division  on  the  particular  set  is  indicated  in  the  parentheses. 


Looking  ahead  once  again,  the  rule  governing  the  frequency  of  execution 


of  the  step  (part  (iii)  of  this  discussion)  shall  call  for  to  leave 


from  each  of  these  subregions.  To  describe  a single  step,  we  shall 


now  stipulate  that  leaves  from  a specific  subregion,  say  • 


This  rule  shall  also  call  for  to  leave  from  one  or  more  points  in 


-»4’  (L  ) • In  the  current  discussion  we  shall  stipulate  that  L is 
P P P 


leaving  from  the  specific  point  3^  c '^^p^^p^ • Also,  we  assigt  the 


reference  time  t to  the  time  at  which  L leaves  ) • 

p P P P 


1 
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We  assume  that  we  have  arrived  at  x at  time  t through  a back- 

-P  P 

ward  trajectory  which  has  been  fashioned  to  satisfy  the  necessary  condi- 
tions. Then  associated  with  x is  some  possibly  infinite  set  of  co- 

-P 

state  vectors,  which  we  shall  designate  A^.  Fortunately,  in  principle 
at  least,  we  shall  be  required  to  deal  with  only  a finite  subset  of 
any  infinite  costate  set.  The  question  of  the  determination  of  this 
subset,  which  remains  partially  unresolved  when  it  contains  more  than 
a single  member,  is  discussed  in  Sections  3. 3. 2. 3 and  3. 3. 3. 2.  We 
assume  for  the  purpose  of  this  discussion  that  this  subset  is  known 
and  that  we  are  dealing  with  a specific  costate  vector  ^ e A^. 

We  now  make  the  following  assumption  regarding  all  optimal  trajec- 
tories propagating  backward  in  time. 

Assumption : No  state  variable  which  is  off  the  boundary  (that  is, 

travelling  on  an  interior  arc)  must  subsequently  return  to  the  boundary 
backward  in  time. 

The  above  assumption  is  certainly  not  always  true,  as  we  demon- 
strate with  a simple  example  in  Section  3. 3.4.4.  However,  in  Section 
4.3  we  show  that  it  is  valid  for  problems  involving  single  destination 
networks  with  all  unity  weightings  in  the  cost  functional. 

Returning  to  our  current  situation,  the  set  of  state  variables 
which  are  travelling  on  interior  arcs  subsequent  to  the  departure  of 

L from  the  boundary  backward  in  time  is  I . = I U L as  indicated  in 
P P-1  P P 

Equation  (3.128).  Also,  the  set  of  state  variables  travelling  on 


-122- 


I 


boundary  arcs  is  8^  = 8 [ L by  (3.130).  According  to  the  assumption, 

all  state  variables  in  I remain  off  the  boundary  for  the  entire 

p-1 

time  interval  T e t ) . We  now  state  the  problem  which  comprises 

P 

the  central  task  of  Operation  1; 

Given  the  state  x and  costate  A at  time  t , find 
-P  -P  P 

all  optimal  trajectories  backward  in  time  on  x £ t^) 


for  whi^.i 


x^(t  ) < 0 
i p - 

x?(T)  = 0 
1 


X^(T)  > 0 


V-  X?  e L 

1 P 

V-  x^  e 8 

1 p-1 

V-  x^  e I , 

1 p-1 


(3.134) 

(3.135) 

(3.136) 


V-  X £ (-«,  t ) 

P 

or  determine  that  no  such  optimal  trajectory  exists. 


Note  that  in  the  above  problem  statement  the  stipulation  of  equa- 
tion (3.134)  guarantees  that  the  state  variables  in  do  in  fact  leave 
(or  stay  on)  the  boundary  backward  in  time  at  t^.  Also,  equation 
(3.135)  expresses  the  desire  to  maintain  on  boundary  arcs  all  of  those 
state  variables  which  are  on  the  boundary  at  t^  and  are  not  members  of 
Lp.  Finally,  equation  (3.136)  is  a formal  statement  of  the  assumption 
stated  above.  Now,  by  the  necessary  and  sufficient  conditions  (Theorems 
3.1  and  3,2  respectively)  we  know  that  any  (and  all)  optimal  trajec- 


tories must  have  a control  which  satisfies  the  following: 
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U*(T)  = ARG  MIN  X^(T)x{T)  = ARG  MIN  (T)B  u(T) 


vdiere 


u (t) eU 


ulT) eU 


A(t  ) = X 

- p -p 


(T)  = -a^  V-  e I , 

111  p-1 


-dX^(T)  = a^dT  + dn^(T) 


dn^(T)  ^ 0 


V-  xr  c 8 


i p-1 


(3.137) 

(3.138) 

(3.139) 

(3.140) 

(3.141) 


Vie  (-“,  t ) . 

p 


Our  tas)^  is  therefore  to  find  all  of  the  solutions  to  (3.137)- 
(3.141)  which  satisfy  the  constraints  (3.134)-(3.136) . The  greatest 
difficulty  is  finding  those  values  of  for  x|  e ^p-1  which  the 
solution  to  (3 . 137) - ( 3. 141)  satisfies  the  constraint  (3.135).  The 
following  simple  proposition  is  helpful  in  resolving  this  dilemma. 

Proposition  3.1;  Any  solution  to  ( 3 . 137) - ( 3 . 141)  which  satisfies 
constraints  ( 3. 134) - ( 3. 135)  is  also  a solution  to 


u*(x)  = ARG  MIN 


X^(t)x^(t) 
u(T)eU  x^tl  , ^ 

1 p-1 


subject  to 


x^ (t")  < 0 

1 p - 


x^(T)  = 0 


V-  X?  € L 

1 P 


¥■  x^  c 8 

1 p-1 


(3.142) 


(3.143) 


(3.144) 
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where 


A? (t  ) = appropriate  component  of  A 


1 P 


A-|(t)  = -a^ 
1 1 


V-  X?  e I , 
1 p-l 


(3.145) 

(3.146) 


V-  T e (-“,  t ) 
p 


Proof:  We  n rewrite  (3.137)  as 


u*(T)  = ARC  MIN 

u(T) eU 


But  by  (3.135)  we  have 


2 Aj(T)xj(T)+  aJ(t)xJ(t) 

xhl  , xhB  , ^ ^ 

i p-l  1 p-l 


c^eB  =0  v-T  e (-»,  t ) 


(3.147) 


(3.148) 


and  therefore  any  solution  to  (3.147)  which  satisfies  (3.135)  is  also 
a solution  to  (3.142).  ■ Proposition  3.1 

We  use  the  following  notation  in  discussing  the  optimization 
problems  presented  above: 


Notation  3.8:  The  optimization  problem  of  ( 3 . 137) - ( 3. 141)  is  referred 

to  as  the  global  optimization  problem.  The  minimand  associated  with  the 
global  optimization  problem, 

i’^rr)x(T)  = A?(t)x^(t), 

x^tl  , ^ ^ xUb  , " ^ 

1 p-l  1 p-l 


(3.149) 
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is  called  the  global  Hamiltonian. 

We  point  out  that  the  Hamiltonian  of  Notation  3.8  differs  from 

T T 

that  defined  in  (3.29)  by  the  additive  term  H ('T))x(T).  How- 

ever, since  x(T)  is  not  an  explicit  function  of  u,  both  are  identical 
with  respect  to  the  pointwise  miminimization. 


j 

i 


Notation  3.9:  The  optimization  problem  (3 . 142) - (3. 146)  of  Proposition 
3,1  is  referred  to  as  the  constrained  optimization  problem.  The  mini- 
mand  associated  with  the  constrained  optimization  problem. 


xhl  , 

1 p-1 


(T)  (T) 


is  called  the  restricted  Hamiltonian. 


(3.150) 


We  repeat  for  emphasis  that  according  to  the  necessary  and  suffi- 
cient conditions  any  (and  all)  optimal  trajectories  must  have  a control 
which  is  a solution  to  t)ie  global  optimization  problem,  that  is,  which 
minimizes  the  global  Hamiltonian.  We  are  looking  for  the  class  of 
optimal  trajectories  which  also  satisfies  the  constraint  (3.144).  Pro- 
position 3,1  tells  us  that  any  (and  all)  optimal  trajectories  in  this 
class  must  also  be  solutions  to  the  constrained  optimization  problem. 
However,  we  must  take  care  to  note  that  solutions  to  the  constrained 
optimization  problem  may  not  be  solutions  to  the  global  optimization 
problem.  See  Section  3. 3.4,1  for  an  example  of  this  behavior.  In  order 
for  a constrained  solution  to  be  a global  solution  there  must  exist 


1 


i 


I 


J 
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values  of  (t)  for  all  £ B , such  that  the  constrained  solution 

1 1 p-1 

satisfies  (3.137)  when  (3.144)  is  released.  These  costates  must 
satisfy  (3.138) - (3. 141) . 

Now,  note  that  we  are  able  to  solve  the  constrained  optimization 
problem  immediately  since  we  know  all  of  the  coefficients  of  (3.142). 
The  values  of  A^(x)  for  £ 8 are  not  required.  The  above  obser- 

1 1 p-1  ^ 

vations  sugg.  it  the  following  two  part  approach  to  finding  all  of  the 
solutions  to  the  global  optimization  problem  which  satisfy  the  con- 
straint equation  (3.135): 

(a)  Find  all  solutions  to  the  constrained  optimization 
problem  ( 3 . 142) - ( 3. 146) . 

' (b)  Produce  values  of  A?(t)  for  all  x^  e 8 , such  that 

i 1 p-1 

the  necessary  conditions  (3.139) - (3.141)  are  satisfied 

I 

and  for  which  all  solutions  to  part  (a)  are  also  solu- 
tions to  the  global  optimization  problem  (3.137)  or_ 
show  that  no  such  values  exist. 

The  above  tasks  were  actually  performed  in  a simple  fashion  for 
the  examples  of  the  previous  section.  Due  to  the  small  dimensionality 
of  the  problems  we  were  actually  able  to  determine  the  globally  opti- 
mizing values  of  the  costates  by  inspection  (such  as  the  determination 
of  in  Excunple  3.1,  case  (i)).  Of  course,  this  is  not  true  in 
general. 

In  Section  3. 3. 3.1  we  present  a method  by  which  the  constrained 
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optimization  problem  of  part  (a)  may  be  solved  parametrically  in  time 
using  linear  programming  techniques.  A geometrical  approach  to  the 
solution  of  part  (b)  is  presented  in  Section  3. 3. 3. 2.  However,  in 
Section  4.3  we  are  able  to  show  that  for  problems  involving  single 
destination  networks  with  all  unity  weightings  in  the  cost  functional 
we  are  always  able  to  produce  the  set  of  costates  X^(T)  for  all 
e which  is  required  in  part  (b)  for  global  optimality.  In 

this  case  part  (b)  is  therefore  unnecessary. 

Suppose  now  that  we  have  managed  to  solve  parts  (a)  and  (b)  for 
the  case  at  hand  and  have  arrived  at  a sequence  of  switching  times  and 
optimal  control  sets  on  (-“,  the  control  encounters  q switches 

over  the  interval  t^)  we  then  denote  the  switching  time  set  as 


T = {t  , . .. , T - , t }. 

p-q  p-1  p 


(3.151) 


In  this  notation  the  control  remains  unchanged  from  t to  minus 

p-q 

infinity.  In  accordance  with  (3.133)  the  collection  of  optimal  control 
sets  is  denoted 


fi  = ...  , , SI  }. 

p-2  p-1 


(3.152) 


Here  the  control  set  S2  is  optimal  from  T to  minus  infinity. 

p-q 

Note  that  we  have  used  t to  denote  the  time  at  which  L leaves 

P P 

the  boundary  and  T ,,  T t to  denote  subsequent  switch 

p-1  p-2  p-q 

times  which  occur  as  r runs  to  minus  infinity.  Although  all  of  the 


above  are  switch  times  in  the  general  sense,  we  give  the  special 
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label  of  boundary  junction  time  since  it  is  forced  on  by  insisting 
that  Lp  leave  the  boundary  at  that  time. 

• Operation  2 

The  purpose  of  this  operation  is  to  utilize  the  sets  T and  fi 

produced  as  output  from  Operation  1 to  construct  feedback  regions  in 
O +p 

n P r -r 

K ^ , where  the  bases  vectors  are  the  elements  of  I , = I U L 

p-1  p p. 

Recall  that  the  trajectory  is  leaving  backward  in  time  from  a point 

X in  a subregion  ) of  m . 

-p  P P "^P 

Suppose  that  proceeding  backward  in  time  the  optimal  control 
encounters  a switch  at  time  T , and  that  the  control  on  [T  , , t ) 

p-1  p-1  p 

is  Then  by  a procedure  to  be  specified  in  Section  3. 3. 3.4  a 

feedback  control  region  is  to  be  constructed  in  R ^ which  has 
r *^p 

i'R  ) as  a wall  in  R We  call  this  region  ^ . and  associate 

P P p-1 

with  it  the  optimal  control  set  The  construction  to  be  specified 

will  assure  us  that  is  the  largest  such  region  with  associated 

control  set 


Notation  3.10;  The  time  ^ is  called  a break  time  of  the  optimal 
trajectory.  If  is  the  state  of  the  optimal  trajectory  at 

then  3^  ^ is  called  a break  point.  Finally,  the  feedback  region 
so  constructed  is  called  a break  feedback  control  region. 


Proceeding  backward  in  time,  if  the  control  switches  at  time  t .,  then 

p-2 

the  break  feedback  control  region  . with  associated  optimal  control 

p-2 
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set  n ^ is  constructed  adjacent  to  . We  continue  in  this  fashion 

p-2  p-1 

to  construct  adjacent  break  regions.  If  the  control  switches  q times, 

then  q break  regions  are  constructed  culminating  in  the  final  break 

region  with  associated  optimal  control  set 

p-q  ^ p-q 

From  time  ^ to  minus  infinity  the  optimal  control  set  does  not 

experience  a break  and  is  In  terms  of  the  construction  of  a 

feedback  -ontrol  region,  this  situation  is  to  be  distinguished  from 

that  of  break  regions,  and  the  construction  technique  is  presented  in 

Section  3. 3. 3. 3.  Applying  this  technique,  we  obtain  a final  feedback 

control  region  ,'/t  adjacent  to  ,■'/}  for  which  the  associated  optimal 

p~g 

control  set  is 

^00 

Notation  3,11:  The  region  is  referred  to  as  a non-break  feedback 

control  region. 

In  summary,  the  sets  T and  fi  are  to  be  utilized  to  construct  a 

a +p 

P P 

total  of  q+1  adjacent  feedback  control  regions  in  R • q break 

regions  and  one  non-break  region.  This  construction  is  illustrated 
in  Figure  3.15. 

At  this  point,  the  reader  may  be  curious  to  know  what  the  various 
regions  described  may  look  like.  We  need  only  refer  back  to  Example  3.2 
to  find  very  simple  examples  of  both  types  of  feedback  regions.  In 
Figure  3.12,  the  portion  of  the  optimal  trajectory  labelled  i2i  - >3) 
corresponds  to  allowing  x^^  leave  from  anywhere  on  the  x^  axis  backward 
in  time.  Since  the  control  switches  at  t^  2'  found  that  for  the 
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Figure  3.15  Construction  of  Successive  Feedback 

Control  Regions  from  Subregion  (i  ) 

P 1 


I 
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cone  bounded  by  the  x^-axis  and  the  line  x^-bx^^  = 0,  not  including  the 
x^-axis,  the  optimal  control  d.s  (3.116).  Hence,  this  cone  is  a break 
feedback  control  region.  Once  the  control  switches  at  it  con- 

tinues on  to  time  equals  minus  infinity  without  a switch.  We  then 


conclude  that  this  control  is  optimal  anywhere  on  the  cone  bounded  by 
the  lines  ~ ® ~ including  the  former  line. 

Therefore  this  cone  is  a non-break  feedback  control  region.  For  this 
simple  example,  we  were  able  to  show  that  for  any  trajectory  of  the 
type  mentioned  above,  the  control  always  switches  on  the  line 
x^-6x^  = 0.  Looking  ahead,  we  shall  generalize  this  notion  by  defining 
a class  of  trajectories  for  which  the  control  always  switches  on  a 
portion  of  a given  hyperplane.  These  surfaces  are  the  o^+p^-l  dimen- 
sional walls  which  separate  the  adjacent  feedback  control  regions 
created  in  Operation  2,  and  shall  be  referred  to  as  breakwalls . 

In  Sections  3. 3. 3. 3 and  3. 3. 3. 4,  we  shall  show  that  the  set 
of  break  walls  and  the  collection  of  optimal  control  sets  12  are  the 
parameters  which  enter  into  the  construction  of  the  feedback  control 
regions  for  a given  backward  optimal  trajectory.  It  is  clear  that 
these  parameters  are  dependent  upon  the  value  of  the  costates  at  the 
point  at  which  leaves  and  therefore  dependent  upon  the  point. 

In  order  to  accommodate  the  constructions  we  now  define  a subregion  as 
follows: 


Definition  3.8:  Suppose  the  set  of  state  variables  in  is  designated 


-132- 


to  leave  the  feedback  control  region  backward  in  time.  Then  a 
subregion  -»i’p(^p)  of  is  the  set  of  all  those  points  in  .^/t^  which 
when  taken  as  the  point  of  departure  of  result  in  a common  and 
common  set  of  breakwalls. 


The  problem  of  determining  subregions  is  confronted  in  Section 
3. 3. 3.2.  However,  once  again  the  special  case  of  single  destination 
networks  with  all  unity  weightings  in  the  cost  functional  provides  us 
with  a simplifying  situation.  In  section  4.3  we  are  able  to  prove  that 
for  this  class  of  problems  there  is  exactly  one  subregion  per  region 
with  respect  to  any  given  L^,  thus  eliminating  the  need  to  determine 
subregions. 


(iii)  Rule  governing  the  frequency  of  execution  of  the  step  and  proof 
that  the  algorithm  realizes  the  constructive  dynamic  programming 
concept 

Suppose  that  some  feedback  control  region  (break  or  non-break) 

0 

has  been  constructed  as  in  Operation  2.  Let  ,/i’  be  a subset  of  R 

where  the  coordinate  axes  of  R ^ are  the  elements  of  the  interior 

state  variable  set  I . Furthermore,  assume  that  ci  < n,  that  is, 

P P P. 

is  a feedback  region  whose  dimension  is  strictly  less  than  the  dimen- 
sion of  the  entire  state  space.  Therefore,  the  set  of  state  variables 
6p  remaining  on  the  boundary  is  non-empty.  We  now  consider  all  subsets 

of  8 which  are  combinations  of  its  elements  taken  1,  2,  ...»  n-o  at 
P « P 

n-0 

a time.  The  total  number  of  such  subsets  is  2 ^-1.  We  then  let  L 

P 
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range  over  all  of  the  subsets  so  determined.  For  each  subset  we  first 
divide  into  subregions  with  respect  to  that  set  as  specified  in 
Definition  3.8.  We  then  perform  the  basic  two  operation  step  of  the 
algorithm  with  the  set  leaving  from  each  of  its  subregions  in  •'A’p* 

Every  step  performed  results  in  a set  of  feedback  control  regions 
constructed  as  part  of  Operation  2.  However,  the  feedback  regions  con- 
structed .t  each  step  may  not  be  new  in  that  they  may  have  been  con- 
structed in  some  previous  step.  We  may  therefore  be  "writing  over" 
regions.  In  essence,  we  are  being  conservative  in  insisting  that 
be  set  equal  successively  to  all  possible  subsets  of  8^,  but  no  method 
is  currently  known  for  the  a priori  elimination  of  those  subsets  which 
will  produce  previously  constructed  regions. 

Let  us  now  view  in  general  terms  what  the  algorithm  is  doing. 

The  algorithm  is  initiated  at  t^  with  the  first  feedback  control  region 
being  the  origin  (which  is  essentially  a degenerate  region)  . The 
set  8^  is  then  composed  of  all  the  state  variables  of  the  problem.  We 
then  allow  to  range  over  all  possible  2*^-1  combinations  of  state 
variables  in  8^;  tliat  is,  we  allow  all  possible  combinations  of  state 
variables  to  leave  the  boundary  backward  in  time  at  t^.  By  Corollary 
3.2  we  know  that  the  values  of  the  costates  at  t^  corresponding  to 
those  state  varicibles  leaving  the  boundary  at  t^  are  zero.  We  are  then 
able  to  solve  the  constrained  optimization  problem  of  this  section. 
(Operation  1,  part  (a))  on  (-^,  t^)  since  all  we  require  are  those 
costates  whicli  correspond  to  state  variables  off  of  the  boundary.  For 
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those  sets  of  state  variables  leaving  the  boundary  which  are  found  to 
have  globally  optimal  trajectories  (Operation  1,  part  (b) ) , feedback 
control  regions  are  constructed  which  range  all  the  way  from  one 
dimensional  (that  is,  axes  of  to  n-dimensional  (that  is,  n-dimen- 

sional  subsets  of  R*')  , We  then  use  each  region  of  this  set  as  the 
starting  point  for  the  sequence  of  steps  which  builds  new  higher  dimen- 
sional regions.  This  process  is  continued  until  all  feedback  control 
regions  constructed  are  n-dimensional. 

Let  us  now  pick  a point  x in  a feedback  region  and  follow  the 
optimal  trajectory  which  takes  the  state  from  5^  to  0^.  Since  is  a 
feedback  control  region,  there  exists  some  control  in  (2^  which  is  opti- 
mal when  applied  at  x.  By  applying  any  control  in  which  keeps  the 
state  in  we  will  eventually  intersect  the  wall  which  separates 

,'/i ^ from  some  adjacent  feedback  control  region  The  trajectory 

will  then  travel  in  with  appropriate  control  in  it 

strikes  the  wall  separating  -^p^2^  from  some  feedback  control  region 
,'/f  _.  We  continue  in  this  fashion  until  the  state  works  its  way 

through  a sequence  of  adjacent  feedback  control  regions  until  it  finally 
reaches  0. 


Proposition  3.2;  Execution  of  the  constructive  dynamic  programming 
algorithm  will  result  in  the  complete  specification  of  the  optimal 
feedback  control. 


Proof : The  rule  governing  the  execution  of  the  step  of  the  algorithm 
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insures  that  feedback  control  regions  are  constructed  for  every  con- 
ceivable type  of  optimal  trajectory  in  terms  of  sequences  of  state 
variables  on  and  off  boundary  arcs.  Moreover,  we  are  finding  the 
largest  such  regions  since  we  are  solving  for  all  optimal  controls 
corresponding  to  each  sequence.  Therefore,  the  complete  execution  of 
the  algorithm  will  result  in  the  entire  space  being  filled  up  with 
feedback  ..ontrol  regions.  ■ Proposition  3.2 

3. 3. 2. 3 Discussion  of  the  Algorithm 

We  have  spoken  in  general  terms  of  constructing  feedback  control 
regions  and  allowing  all  possible  sets  of  state  variables  remaining 
on  the  boundary  to  leave  from  these  regions  backward  in  time.  However, 
the  reader  should  not  lose  sight  of  the  fact  that  we  are  employing  a 
specific  set  of  optimal  trajectories  constructed  backward  in  time  to 
satisfy  the  necessary  conditions.  In  this  section  we  take  a look  at 
a typical  backward  optimal  trajectory  and  enumerate  some  of  the  basic 
problems  involved  in  the  execution  of  the  algorithm.  Specifically, 
we  establish  the  notational  setting  in  which  these  problems  are  to  be 
attacked  in  subsequent  sections. 


Global  Optimality 

Consider  the  situation  in  which  we  allow  the  state  varial)les  in 
to  leave  the  boundary  at  some  boundary  junction  time  This 

situation  is  illustrated  in  Figure  3.16.  We  defer  the  question  of  how 


L36- 


I 


^p-q  • • • 'p  '■p+1 


Figure  3.16  Globally  Optimizing  Costate  Trajectories 
for  Given  State  Trajectory 
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boundary  junction  times  are  determined  until  the  discussion  under  the 
heading  of  "subregions"  below.  Also,  we  assume  that  we  are  given  a 
particular  value  of  ^p+1  ® potentially  non-unique 

set  appropriate  costates  at  that  time.  See  Figure  3.16.  We 

shall  specify  what  we  mean  by  "appropriate"  in  the  discussion  under 
the  heading  of  "leave-the-boundary  costates"  below. 

We  do  not  know  a priori  if  the  situation  described  above  is 
optimal.  We  begin  by  calculating  all  optimal  solutions  on  (-“, 
via  the  constrained  optimization  technique  of  Operation  1,  part  (a). 
This  is  illustrated  for  a particular  trajectory  in  Figure  3.16  . Once 
this  is  accomplished,  we  must  satisfy  part  (b)  of  Operation  1 by  pro- 


ducing a legitimate  set  of  costates  on  (-“,  t .)  for  which  the  solution 

p+1 

on  this  interval  is  globally  optimal.  To  be  more  specific,  since  A^(I) 

for  X?  e are  determined  exactly  by  (3.139)  for  the  given  Ap+i' 

task  is  actually  to  produce  a set  of  costates  A^(x)  for  e 8^  which: 

(i)  satisfy  the  necessary  conditions  and  (ii)  together  with  A^(x)  for 

x^  e J result  in  a globally  optimal  solution  on  (-“,  t ,). 

1 P p+1 

We  shall  now  specify  all  of  those  values  of  X^(x)  for  x^  e 8 

lip 

which  satisfy  the  necessary  conditions.  At  time  t , we  have  assumed 

p+1 

that  we  are  given  a specific  costate  value  X from  among  a potentially 

1 

non-unique  costate  set  A , ; that  is,  the  values  of  X? (t  ,)  for  all 

p+1  1 p+ 1 

x^  £ 6p  are  given.  If  we  designate  x to  be  time  running  backward  from 
then  equations  ( 3 . 140)  - ( 3. 141)  indicate  that  the  maximum  value 
any  x|(x)  may  achieve  is  when  we  take  dr)|(x)  =0.  In  this  case  we  have 
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Figure  3.17  Possible  values  of  when 

I 1 

is  on  boundary  arc  (x^  = 0) . 

Therefore,  as  a function  of  time  the  value  of  X^(t)  for  x^  e 8 which 

1 1 p 

satisfy  the  necessary  conditions  are  those  which  fall  below  (t  ,) 

1 p+1 

+ Ta^-  This  is  depicted  in  Figure  3.17,  We  are  free  to  pick  any  of 
these  values  in  our  attempt  to  satisfy  global  optimality  on 

Finally,  we  note  that  part  (b)  of  Operation  1 requires  only  that 
we  produce  a single  set  of  costates  for  which  the  solution  is  globally 


optimal.  However,  there  may  be  a non-unicjue  class  of  such  costates 


which  are  legitimate,  as  indicated  in  Figure  3,16. 
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Leave-the-boundary  costates 


If  the  solutions  obtained  in  Operation  1 are  globally  optimal, 
then  we  proceed  to  construct  from  these  solutions  a set  of  feedback 
control  regions  in  Operation  2.  The  details  of  the  constructions  are 
presented  in  Sections  3. 3. 3. 3 and  3. 3. 3.4.  Suppose  now  that  we  have 
completed  Operation  2 and  that  the  feedback  control  regions  which  have 
been  cons  tructed  are  . Here  we  have 

^ p-q  p-i  p 

assumed  that  the  control  switches  q times  on  that  the 

switch  time  set  and  control  set  are  as  in  (3.151)  and  (3.152)  respec- 
tively. Therefore  is  constructed  from  the  solution  parameters  on 
[t  , t , ) , -*>  , is  constructed  from  the  solution  parameters  on 

p p+1  ‘p-1 

(t  ,,  T ),...,  ./?  is  constructed  from  the  solution  parameters  on 

p-1  p -00 

(-°°,  T ).  Recall  that  is  a non-break  feedback  control  region  and 

p-q  -00 

all  of  the  rest  , t ,‘/f  ) are  break  feedback  control 

p-q  p-1  p 

regions. 

Now,  the  rule  governing  the  frequency  of  execution  of  the  step 

of  the  algorithm  calls  for  us  to  let  all  possible  combinations  of 

state  variables  in  B to  leave  backward  in  time  from  each  of  these 
P 

regions.  As  an  example,  let  us  restrict  attention  to  the  particular 


feedback  control  region  and  a particular  set  of  state  variables 

L C B • Allowing  the  state  variables  in  L to  leave  backward  in  time 
P P P 

from  some  point  in  is  equivalent  to  allowing  them  to  leave  at 

some  boundary  junction  time  t^^,  where  t^^  falls  on  the  interval 

[x  , t ,).  In  order  to  allow  tlie  state  varialJles  in  L to  leave 
P P+1  P P 
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at  we  must  perform  the  following  task: 

Determine  the  values  of  A^(t  ) for  e B which 

1 p r P 

allow  to  leave  the  boundary  backward  in  time  optimally 
for  all  potential  boundary  junction  times  t^  E [1^, 
or  show  that  no  such  values  exist  which  satisfy  the 
necessary  conditions. 


These  values  may  be  nonunique  and  will  be  a subset  of  the  glo- 
bally optimizing  values  as  illustrated  in  Figure  3.18.  Now,  the 

costates  so  determined  constitute  A and  will  serve  as  initial  condi- 

P 

tions  for  the  determination  of  the  backward  optimal  trajectory  which 

must  be  solved  for  the  state  variables  in  leaving  from  the  feedback 

control  region  . We  refer  to  the  above  task  as  the  determination 
P 

of  leave-the-boundary  costates  at  t^,  where  the  implication  is  that 

the  costates  are  such  that  they  allow  the  states  in  L to  leave  the 

P 

boundary  optimally  at  t^. 

Subregions 

The  required  division  of  into  subregions  with  respect  to  L 

P P 

is  equivalent  to  dividing  [t  , t ) into  a set  of  time  sub intervals . 

P P+-'- 

Over  each  subinterval,  the  subsequent  optimal  control  sets  and  break- 


walls  on  (-UJ,  tp)  corresponding  to  the  state  variables  in  leaving 
the  boundary  are  the  same  for  any  point  on  the  subinterval,  by  Defini- 
tion 3.8.  The  values  of  A^(t  ) for  x^  c 8 are  drawn  from  Uie  leave- 

1 p 1 P 
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the-boundary  set  discussed  above.  Since  may  be  infinite,  we  must 
somehow  reduce  to  a finite  number  those  values  which  must  be  considered 
in  this  fashion.  This  problem  is  discussed  in  Section  3. 3. 4.2.  In 
Section  3. 3.4.3  we  discuss  the  issues  involved  in  performing  the  sub- 
division for  a particular  A e A . The  time  subdivision  which  we  seek. 

-P  P 

is  illustrated  conceptually  in  Figure  3.18,  which  is  an  enlarged  pic- 
ture of  the  iterval  [T  , t , ) . In  this  case  there  are  s subintervals 

P P+1 

corresponding  to  s subregions  with  respect  to  L^.  We  use  the  symbols 

t|,  T^,  ..,  ^ to  designate  the  extremities  of  the  subintervals  on 

[T  , t ),  with  the  exception  of  T and  t The  subinterval 

P P+1  P P+1 

[T' , t corresponds  to  the  subregion  {I  ),  the  subinterval 
1 p+1  P p 


[T2,T|)  to  the  subregion  •••'  ’^'^bil  finally  the  subinterval 


[T  , p)  corresponds  to  the  subregion  ). 


P s-i  * P P 

We  clearly  have  many  questions  to  answer  in  order  to  be  in  a posi- 
tion to  perform  the  algorithm.  The  basic  procedures  are  confronted  in 
Section  3.3.3  and  complicating  features  are  discussed  in  Section  3.3.4. 


3.3.3  Basic  Procedures  of  the  Algorithm 
3. 3. 3.1  Solution  of  Constrained  Optimization  Problem 
The  first  task  of  Operation  1 is  the  solution  of  the  constrained 
optimization  problem  ( 3 . 142) - ( 3. 146)  pointwise  in  time  on  the  interval 
T f.  (-“,  tp)  . We  begin  by  expressing  the  program  in  terms  of  the 
underlying  decision  variables,  which  are  the  controls,  and  also  by 
integrating  (3.146)  backward  in  time: 
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u*(T)  = ARG  MIN  ^ X^(T)[b?  u(t)  + ] 

u(T)eU  x^el  ^ 

- i p-1 


(3.154) 


subject  to 


b^ 
— 1 

U(t") 
~ p 

< -a^ 

— 1 

V £ L 

1 p 

(3.155) 

— 1 

u(x)  = 

1 

¥ x^  e 8 

1 p-1 

(3.156) 

= 

X^(t  ) 

1 p 

+ xa? 
1 

¥ x^  e I 

i p-1 

(3.157) 

T e (-“,  t ) 

P 

where  X is  time  measured  backward  from  t^  and  b^  represents  the  row  of 
^ corresponding  to  x?.  We  re-write  (3 . 154) - ( 3. 157)  by  first  recog- 
nizing that  in  (3.154)  is  a constant  additive  term  which  does  not 
affect  the  minimization  and  also  be  substituting  (3.157)  into  (3.154) 
to  obtain 


u*(X)  = ARG  MIN  (^  + TC  )B  u(x) 


u(T)eU' 


■1 


(3.158) 


where 


D u(T)  ^ C 
u (X ) ^0 

b?u(o")  < -a?  V-  X?  e L 
—1—  — 1 ^ P 

b^u(x)  = -a?  V-  x^  e 8 

—1—  1 1 p-1 


(3.159a) 

(3.159b) 

(3.159c) 

(3.159d) 


= \^(0)b^ 


cYei 

1 p-1 


(3.160) 


1 
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(3.161) 


c = 2^ 

i p-1 

T e (0,  . 


Here  we  have  simply  set  = 0.  Equations  (3. 158) - (3. 161)  represent 
the  final  form  of  the  program  with  which  we  shall  be  dealing.  This 
program  differs  from  the  global  minimization  in  that  we  have  the  addi- 
tional constraints  (3.159c)  and  (3.159d).  Hence,  the  solution  set 
specified  by  the  control  law  (3.41) -(3.43)  is  too  large  for  our  purpose. 
Unfortunately,  the  additional  constraints  preclude  the  solution  of  the 
program  directly  as  a function  of  the  costates  as  in  (3.41) -(3.43) . 

However,  since  for  fixed  T (3. 158) -(3.161 ) is  a linear  program, 
the  Simplex  technique  may  be  applied  to  find  a solution.  Moreover,  the 
cost  function  of  (3.158)  is  a linear  function  of  the  single  independent 
parameter  T,  while  the  constraints  are  not  a function  of  T since  £ is 
constant.  This  is  precisely  the  form  which  can  be  accommodated  by 
parametric  linear  programming  with  respect  to  the  cost  coefficients. 

The  solution  proceeds  as  follows: 

Set  T = 6,  where  6 is  some  small  positive  number  which  serves  to 
perturb  all  costate  values  by  . We  wish  to  start  our  solution  at 
time  t -6  since  we  may  have  X^(t  ) =0  for  some  x^  e 8 , , so  that  the 


1 P 


solution  exactly  at  t^  may  not  correspond  to  x^  leaving  the  boundary. 
Also,  constraint  (3.159c)  is  active  at  this  point  to  insure  that  the 
state  boundaries  x^  > 0,  x^  c L , are  not  violated  just  after  t (in 

X - 1 p p 
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backward  time) . The  number  6 must  be  such  that  0 < 6 < T , , where 

p-1 

T , is  the  first  break  time  to  be  encountered  backward  in  time. 

p-1 

We  now  use  the  Simplex  technique  to  solve  the  program  at  T =6. 
There  are  many  linear  programming  computer  packages  which  may  be  en- 
listed for  this  task  which  utilize  efficient  algorithmic  forms  of  the 
Simplex  technique  to  arrive  at  a single  optimal  extremum  solution. 

Given  thi 3 starting  solution  which  we  call  ^ most  packages  are 
also  equipped  to  employ  parametric  linear  programming  to  find  the  value 
of  T for  which  the  current  solution  ceases  to  be  optimal  as  well  as  a 
new  optimal  solution.  These  are  the  break  time  ^ and  the  optimal 
control  ^ respectively.  We  continue  in  this  fashion  to  find  con- 
trols and  break  times  until  the  solution  remains  the  same  for  T arbi- 
trarily large.  This  final  solution  is  the  control  u_j^,. 

The  linearity  of  the  pointwise  minimization  associated  with  the 
necessary  conditions  has  enabled  us  to  find  a sequence  of  optimal 

controls  on  the  time  interval  (-“,  t ) by  the  efficient  technique  of 

P 

parametric  linear  programming.  However,  in  the  description  of  Opera- 
tion 1 in  Section  3. 3.2.2  we  call  for  all  optimal  solutions  on  every 
time  segment.  Since  we  are  dealing  with  a linear  program,  the  speci- 
fication of  all  optimal  solutions  is  equivalent  to  the  specification 
of  all  optimal  extremum  of  the  solution  set.  Unfortunately,  it  turns 
out  that  the  problem  of  finding  all  the  optimal  extremum  solutions  to 
a linear  program  is  an  extremely  difficult  one.  In  Appendix  A we  show 


that  given  an  initial  optimal  extremum  solution  this  problem  is  equiva- 
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lent  to  finding  all  the  vertices  of  a convex  polyhedral  set  defined  by 
a system  of  linear  equality  and  inequality  constraints.  Discussion  of 
this  problem  has  appeared  intermittently  in  the  linear  programming 
literature  since  the  early  1950'e,  where  several  algorithms  based  upon 
different  approaches  have  leen  presented.  See  for  example  Balinski 
[1961]  and  Mattheiss  [1973] . However,  none  of  these  methods  has  proven 
computationally  efficient  for  a reasonably  large  variety  of  problems. 
The  fundamental  difficulty  which  appears  to  foil  many  algorithms,  no 
matter  what  their  underlying  approach,  is  degeneracy  in  the  original 
linear  program.  As  our  problem  is  characterized  by  a high  degree  of 
degeneracy,  one  would  expect  poor  performance  from  any  of  these 
algorithms . 

In  Appendix  A we  describe  an  algorithm  developed  by  Bloom  [1976] 
in  conjunction  with  this  research  for  the  solution  of  the  problem  of 
finding  all  of  the  extremum  solutions  of  a linear  program.  It  is 
based  upon  an  algorithm  of  Chemikova  [1965]  which  finds  the  extreme 
rays  of  a convex  cone  defined  by  linear  equality  and  inequality  con- 
straints. By  a simple  transformation,  this  algorithm  can  be  adapted 
to  find  the  vertices  of  a convex  polyhedral  set,  which  is  the  problem 
to  be  solved.  Limited  experience  with  the  application  of  the  algorithm 
indicates  that  the  amount  of  computation  and  storage  involved  may  be 
excessive.  These  problems  are  magnified  by  the  extreme  numerical  sen- 
sitivity of  the  algorithm. 

It  appears  at  this  time  that  the  development  of  an  efficient 
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algorithm  for  the  solution  of  this  problem  is  contingent  upon  the  dis- 
covery of  methods  for  resolving  degeneracy  in  linear  programs.  As 
degeneracy  is  a fundamental  nuisance  in  most  linear  programming  pro- 
cedures, this  problem  is  the  subject  of  much  ongoing  research. 

At  any  rate,  assuming  that  all  optimal  extremum  may  be  found  for 
every  time  segment,  the  output  of  this  procedure  consists  of  the  break 
time  set  '3.151)  and  the  collection  of  optimal  control  sets  (3.152). 
Note  that  is  now  considered  to  be  the  set  of  all  optimal  extremum 
solutions  to  the  constrained  linear  program  on  X £ [t  , t ,).  In 

p p+1 

Sections  3. 3. 3.3  and  3. 3. 3.4  the  requirement  that  we  find  all  optimal 
extremum  is  justified. 

3. 3. 3. 2 Determination  of  Leave- the-Boundary  Costates, 

Subregions  and  Global  Optimality 
In  this  section  we  introduce  a geometrical  notion  which  serves 
as  a setting  for  understanding  and  evaluating  various  issues  of  the 
algorithm  which  were  identified  in  Sections  3. 3. 2. 2 and  3. 3. 2. 3. 

Preliminaries 

We  begin  by  considering  the  basic  linear  program  (3.37)  of  the 

necessary  conditions  to  be  solved  pointwise  in  time.  The  elements  of 

the  m-dimensional  control  vector  u(x)  comprise  the  underlying  set  of 

decision  variables  in  R*".  However,  since  the  cost  functional 
T • , 

^ (t)3^(t)  is  explicitly  linear  in  >^,  it  is  quite  natural  to  view  the 
activity  of  the  linear  program  in  the  space  R^  with  the  decision 
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vector  X.  We  begin  by  denoting 

Jl(T)  = -x(T)  = -B  u(T)  - a (3.162) 

where  the  negative  sign  has  been  introduced  solely  for  the  convenience 
of  discussion. 

Definition  3.9;  We  denote  by 

;'/  = 1;^  e = "2.  H.  “ £ u e (i}  c R"  (3.163) 

the  set  of  negative  feasible  flows  attainable  through  the  admissible 
controls  in  the  presence  of  constant  inputs  a.  We  shall  refer  to 
as  the  ^-constraint  figure. 

Note  that  the  only  difference  between  as  defined  above  and.i^' 
as  defined  in  (3.44)  is  that  the  former  accounts  for  the  inputs  a; 
therefore,  M is  obtained  from  through  simple  translation  by  the 
vector  a.  As  such,  is  also  a bounded  convex  polyhedron  in  We 

may  now  state  the  representation  of  the  pointwise  linear  program  (3.37) 
as  it  appears  in  R*^; 

il*(T)  = ARG  MAX  u'^(T)il(T)]  (3.164) 

j[.(t)  e 

T c [t^,  t^] . 

Note  that  the  maximum  has  been  tal^en  in  (3.164)  as  opposed  to 
tile  minimum  in  (3.37)  due  to  the  introduction  of  the  negative  sign  in 
(3.162).  The  solution  ^*(t)  to  (3.164)  represents  the  negative  of  the 
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optimal  state  rate  x*(t)  = ^ u*(T)  + a at  every  point  in  time.  How- 
ever, we  shall  not  be  interested  in  solving  (3.164)  directly  since  in 
general  we  do  not  have  an  explicit  set  of  linear  constraints  which 
comprise  .'V.  Instead,  we  shall  be  interested  in  the  underlying  pro- 
perties of  the  linear  program  as  it  is  expressed  in  (3.164). 

T 

If  we  consider  time  to  be  fixed  and  express  z = ^ ^ as  the  objec- 
tive func‘-ion  of  (3.164),  then  the  geometrical  characterization  of 

T 

linear  programs  calls  for  the  equation  z = ^ ^ to  be  visualized  as  an 

n-1  dimensional  hyperplane  in  R^.  For  a fixed  value  of  varying 

T 

the  value  of  z causes  the  hyperplane  z = ^ ^ to  translate  parallel 
T 

to  the  hyperplane  ^ ^ = 0-  The  optimal  solution  (solutions)  is 
T 

achieved  for  z*  = X tangent  to  the  constant  set  . The  solution 
set  consists  of  all  the  points  of  tangency,  and  may  range  from  a single 
vertex  of  to  an  n-1  dimensional  face  of  .'/. 

The  geometrical  interpretation  is  depicted  in  Figure  3.19. 

The  linear  program  (3.164)  is  the  ^-space  version  of  the  global 
optimization  (3.37).  However,  in  part  (a)  of  Operation  1 of  the 
algorithm  we  are  dealing  with  the  constrained  optimization  problem. 

We  now  proceed  to  express  this  problem  as  it  appears  in  ^-space.  First 
the  notation: 

Notation  3.12 


z(T) 


(3.165) 


/ 

increasing  z 


solution  set 


Figure  3.iy  Geometrical  Interpretation  of  Pointwise 
Linear  Program  in  ^-space 


is  the  restricted  Hcuniltonian  at  time  t.  Geometrically,  it  is  the 
o -1  dimensional  restricted  Hamiltonian  hyperplane  at  time  T.  We  denote 
this  hyperplane  as  H^(t) . 

Notation  3.13 

0 . . o 

= {y  e R ^|y  E.  y/  and  y?  = 0 V-x^cSli'R^  (3.166) 

p^  1 ip 

is  the  a dimensional  restricted  y^-constraint  figure. 
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I 


[i 


The  constrained  optimization  problem  in  ^-space  is 


^*(T)  = ARG  MAX  z(T) 
il(T)  e 

T e (-«>,  t ) . 

p 


(3.167) 


The  solution  to  (3.167)  is  piecewise  constant  on  time  segments  identical 
to  those  for  the  underlying  u-space  problem.  We  therefore  use  similar 
notation  for  specifying  the  solutions. 


Notation  3.14 


y = a solution  to  (3.167)  on  [T  , i ^.) , referred  to 
TJ  P P+1 

as  a Op-dimensional  operating  point 
y = the  set  of  all  solutions  to  (3.167)  on  [T  , T ,). 

p P P+1 

y = {y^,  ...»  sequence  of  the 

solution  sets  to  (3.167)  on  (-“,  t^)  . 


(3.168) 


We  now  demonstrate  the  above  concepts  with  the  aid  of  a familiar 
excunple . 


Example  3.3 

Take  the  network  and  cost  functional  of  Example  3.1.  Then  the 
global  constraint  figure  is 


= {yi-yslvi  = ^13^^12’'^21'  '^2  = ^23“'^12^^21' 

°i'^23-^'°^‘ 


(3.169) 
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This  is  illustrated  in  Figure  3.20. 

We  now  follow  on  Figure  3.20  the  steps  of  the  algorithm  associated 
with  situation  (i)  of  Example  3.2.  First,  we  let  that  is, 

Xj^  leaves  the  boundary  at  t^.  The  sought  after  solution  must  have  x^^ 
negative  and  X2  = 0.  This  corresponds  to  the  constrained  maximization 


= ARG  MAX  X (T)y  (T) 


T e:  [t^_^,  t^] 


^13'^“l2”“21 


0<U2ifp.5,  0<Uj^3£1.0}, 


(3.170) 


(3.171) 


The  one  dimensional  restricted  constraint  figure  is  illustrated 

by  the  heavy  line  in  Figure  3.20,  along  with  the  restricted  Hamiltonian 
H^_j^(T):  z(T)  = X^(T)y^.  Clearly,  the  maximum  of  over  is 

achieved  at  the  one  dimensional  operating  point  ^ = (1.5,  0)^.  This 

point  is  globally  maximizing  if  and  only  if  X^d)  = 0.  The  solution 
point  corresponds  to  the  control  (3.93)  and  allows  us  to  specify 

the  feedback  control  on  the  Xj^-axis  as  in  Figure  3.9. 

Next,  we  set  ^2  leave  the  boundary 

backward  in  time  at  . Since  all  states  are  now  in  the  interior, 
the  maximization  is  global: 

^*(T)  = ARG  MAX  [X  (T)y, (T)+X  (T)y  (T) ) (3.172) 

i.(T)e  ,'/^_2=// 
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The  two  dimensional  global  constraint  figure  2 “ is  illustrated 

in  Figure  3,20,  along  with  the  global  Hamiltonian  H^_2(t):z(T)  = 

Xi(T)yi  + A2(T)y2«  Since  the  costate  behavior  as  specified  in  (3.96) 

and  (3.97)  implies  X^^d)  > X2(T),  the  two  dimensional  operating  point 
T 

is  ^_2  ~ (1-5,  0.5)  . This  point  corresponds  to  the  control  (3.98). 
This  is  easij-y  seen  from  Figure  3.20,  As  time  proceeds  bac)cward,  the 
relative  values  of  the  costates  cause  the  Hamiltonian  H^  ^ rotate 
in  the  direction  indicated  by  the  arrow.  As  time  approaches  minus 
infinity,  H^_2  approaches  the  face  of  which  lies  on  the  line 
y^+y2  = 2.0.  Therefore,  there  is  no  breaJt  point  in  the  control  as  time 
approaches  minus  infinity.  Finally,  ^ ^ is  used  to  construct  the 
shaded  non-break  feedback  region  of  Figure  3.9.  •Example  3.3 


Geometrical  Interpretation 

The  observations  of  the  preceding  examples  suggest  the  following 
geometrical  interpretation  of  the  backward  constructive  algorithm  pre- 
sented in  Section  3.3:  Suppose  the  set  of  state  variables  in 

leaves  the  boundary  backward  in  time  at  resulting  in  the  set  of 

state  variables  I on  interior  arcs.  Then  the  constrained  optimization 

o 

problem  of  Operation  1,  part  (a),  takes  place  in  R where  the  basis 

a 

vectors  of  R ^ are  the  elements  of  I . The  constraint  set  is 

P 


//  = n R 
p 


\ 

1 


(3.173) 
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where  ,y  is  the  global  constraint  set  in  defined  by  (3.163).  The 
objective  function  is  represented  by  the  dimensional  restricted 

Hamiltonian  hyperplane  in  R 


H (T)  jz(T) 
P 


hi  ^ 


(3.174) 


T e (- 


4.1^* 

P+1 


The  initial  solution  set  of  the  linear  program  consists  of  all  points 

of  tangency  between  and  H (T)  on  T £ [T  , T ) . This  solution  set 

p p P P+i 

consists  of  one  or  more  extreme  points  of  and  all  the  points  which 

are  convex  combinations  of  the  extreme  points.  Each  such  point  is  a 

0^-dimensional  operating  point.  Note,  however,  that  an  extreme  point 

of  is  not  necessarily  an  extreme  point  of  (e.g.  r ^ 

Figure  3.20  is  an  extreme  point  of  ^ but  only  a boundary  point  of 

) . Now,  as  time  runs  baclcward  toward  minus  infinity,  the  costates 

evolve  according  to  (3.157)  in  general  causing  the  orientation  of  the 

hyperplane  H (t)  to  change  with  respect  to  //  . If  H (x)  moves  a 
P P P 

sufficient  amount,  then  the  surface  of  tangency  with  may  change. 

This  occurrence  defines  the  break  time  T and  the  solution  set  will 


P 

change  to  the  new  points  of  tangency.  This  will  continue  until  some 
solution  set  persists  to  time  equals  minus  infinity.  This  version  of 
the  constrained  optimization  problem,  which  represents  part  (a)  of 
Operation  1 of  the  algorithm,  is  illustrated  in  Figure  3.21.  For 
the  purpose  of  this  discussion,  let  us  now  assume  that  the  solutions 


H (0 
P 


T t:  (-00, 


Vl' 


Fi.jiire  J.21 


Geometrical  Interpretation  in  y-space 
oi  Operation  1,  part  (a). 
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obtained  satisfy  part  (b)  of  Operation  1,  that  is,  they  are  globally 
optimal. 


Suppose  that  we  now  let  the  set  of  state  variables  leave  the 

boundary  some  time  on  the  interval  [T  , t , , ) at  boundary  junction 

P P+-*- 

time  tp.  Then  the  constrained  optimization  problem  takes  place  in 
O +p 

R ^ where  the  basis  vectors  are  the  coordinates  of  I . = I U i , 

p-1  p p 


The  constraint  set  is  now 


a +p 

,-y  = ,v/  n R P P. 

p-1 


(3.175) 


The  objective  function  is  represented  by  the  O +p  -1  dimensional  re- 

a +p 

stricted  Hamiltonian  hyperplane  in  R P P: 


H (T)  :z(T)  = y]  (T)y^ 

1 p-1 


(3.176) 


T e (^,  tp). 

Once  again,  solutions  to  the  linear  program  consist  of  all  points 
of  tangency  of  H , (T)  with  See  Figure  3.22.  These  solutions 

p-1  p-1 

consist  of  one  or  more  extreme  points  of  points  which 

are  convex  combinations  of  these  extreme  points.  These  points  are 
O +p  dimension  operating  points  of  .'V  • We  repeat  for  emphasis  that 

they  are  not  necessarily  extreme  points  of  •'>.  As  we  let  time  run  to 
minus  infinity,  H (T)  rotates  and  picks  up  new  solution  sets  until 

p-1 

finally  one  persists  to  minus  infinity.  For  the  moment  we  continue  to 


assume  that  all  constrained  solutions  obtained  are  also  globally 


optimal . 
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We  have  now  described  in  geometrical  terms  the  nature  of  suc- 
ceeding constrained  optimization  steps  of  the  algorithm.  In  summary, 
as  we  allow  state  variables  to  leave  the  boundary  backward  in  time,  we 
continually  enlarge  the  space  of  free  decision  variables  by  releasing 
constraints  of  the  form  y^  = 0.  As  the  constraint  figure  grows  in 
dimension,  so  also  does  the  associated  Hamiltonian  hyperplane.  When 
all  the  s^-^te  variables  have  finally  left  the  boundary  backward  in 
time,  the  constraint  figure  is  the  n-dimensional  figure  and  the 
Hamiltonian  is  represented  by  an  n-1  dimensional  hyperplane. 

The  geometrical  interpretation  introduced  above  shall  now  be 
applied  to  discuss  in  detail  the  basic  issues  presented  in  Section 
3. 3. 2. 3 (global  optimality,  leave-the-boundary  costates,  subregions). 
We  begin  by  establishing  some  definitions  and  a theorem  which  charac- 
terize an  important  geometrical  relationship  between  successive  con- 
strained optimization  problems. 


Geometrical  Relationship  between  Successive  Constrained 
Optimization  Problems 

Suppose  we  have  a set  of  state  variables  interior  arcs  and 

that  the  set  of  Op-dimensional  operating  points  is  Y^.  At  some 

boundary  junction  time  t^  c [1^,  wish  to  allow  the  set  of 

state  variables  in  the  set  ^p  leave  the  boundary  backward  in  time, 

where  L C 5 . Consider  now  the  relationship  between  the  O dimensional 
p p P 

constrained  optimization  problem  at  t and  the  o +p  dimensional  con- 

P P P 
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strained  optimization  problem  at  t^.  We  begin  by  noting  that  equations 
(3.173)  cind  (3.175)  together  imply 


,y  = fy  , ^ R ^ 

p p-i 


(3.177) 


Therefore,  all  boundary  points  of  are  also  boundary  points  of 
In  particular,  extreme  points  of  are  boundary  points  of  i’ 

Definition  3..^:  Consider  the  set  of  operating  points  Y C 3^  . Let 

P P P 

R ^ be  the  p dimensional  space  whose  basis  vector's  are  the  elements 
P 

of  L . "hen  we  call  any  face  oi  ..  , which  contains  the  set  Y and 

P P-1  P 

contains  at  least  one  point  in  the  positive  orthant  of  R an 

L -positive  face. 




Most  central  to  this  definition  is  that  by  (3.162)  any  point  in 

the  positive  orthant  of  R ^ has  strictly  negative  for  all  x^  e 

(recall  that  this  is  in  the  preseiice  of  inputs  £)  . Since  we  intend  for 

the  state  variables  x?  e L to  leave  the  boundary  backward  in  time,  then 
1 £ [ 

• j 

we  require  x^  strictly  negative  (in  forward  time).  To  illustrate  the 

notion  of  Definition  3.9,  we  refer  to  Figure  3.22  as  an  example.  In 

this  case,  the  L -positive  faces  of  //  , with  respect  to  Y are  those 

P P-1  P 

labelled  F^^,  F^  and  F^  in  the  figure. 

The  major  result  of  this  section  is  contained  in  the  following 


theorem.  Here  we  provide  the  link  between  the  a -1  dimensional  Hamil- 

P 

tonian  hyperplane  H (t^)  and  the  o +p  -1  dimensional  hyperplane 
P P P P 

Hp_j^(tp)  which  has  the  state  variables  in  leaving  the  boundary  back- 
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ward  in  time  at  t . 

P 

Theorem  3 . 5 

Suppose  at  time  t the  current  set  of  operating  points  is  y C 

P P P 

for  the  restricted  Hamiltonian  H (t  ) . Let  L be  the  set  of  state 

P P P 

variables  to  leave  the  boundary  at  t^ . Then  any  (possibly  restricted) 
Hamiltonian  H , (t  ) which  allows  L to  leave  the  boundary  at  t is 

p-1  p p p 

such  that: 

p 

(a)  H (t  ) = H (t  ) n R P 

P P P-1  P 

(b)  H , (t  ) contains  a L -positive  face  of 

p-1  p p ^ p-1 

P 

where  R ^ is  as  defined  in  Definition  3.9. 


Proof: 


(a)  The  restricted  Hamiltonian  hyperplane  H^  at  the  time  just 

prior  to  the  departure  of  L is 

P 


H (t^):z(t^)  = X^(t^)y?. 

P P P ^j,J  X p'^'i 


(3.178) 


The  (possibly  restricted)  Hamiltonian  hyperplane  H^_j^  at  the  time  just 

following  the  departure  of  L has  the  form: 

P 


H (t  ):z(t  ) = S X^(t  )y^  + X^(t  )y^. 

P-1  P P X p^  3^  X pVx 


(3.179) 


x-:ci 
1 P 


x^cL 
X p 


Now,  H^(t^)  contains  all  jxDints  in  Y^.  Also,  since  we  are  assuming 
Yp_j^  is  such  that  the  state  variables  in  leave  the  boundary,  there 
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must  be  at  least  one  point  i!p  ^ ^ which  > 0 for  all 

X?  e L . That  is,  the  slope  in  forward  time  associated  with  x?  e L 

X p X p 

must  be  strictly  negative  in  order  that  leave  the  boundary  backward 

in  time.  If  we  now  evaluate  H (t^)  at  any  point  y e Y and  H , (t  ) 

p p P p-1  P 

at  the  point  ^ ^p-1'  condition  (3.14)  , which  stipulates 

continuity  of  the  Hamiltonian  everywhere,  gives 


z (t  ) = z(t  ) = z(t  ) . 
P P P 


(3.180) 


Furthermore,  we  have  that 


X^(t'^)  = X^(t“)  = ) V- e I 

1 p 1 p'  1 p'  1 P 


(3.181) 


by  equation  (3.32).  By  virtue  of  (3.180)  and  (3.181)  we  may  write  the 


Hamiltonians  as 


H (t  ):z(t^)  = Aj(t  )y^ 

P P P 1 P 


(3.182) 


H .(t  ):z(t  ) = .5^  A^(t  )y^  + A?(t  )y^. 

P P x^el  " P " xhL  " P " 


(3.183) 


We  conclude  immediately  from  (3.182)  and  (3.183)  that  H (t  ) 

P P 


H , (y  ) n R"P. 
P-1  P 


(b)  From  part  (a)  we  see  that  H , (t  ) contains  H (t  ) and 

p-1  p p p 

therefore  contains  Y . Also,  11  , (t  ) contains  at  least  one  point 

p p-1  p ^ 

Pp 

V . in  the  positive  orthant  of  R Therefore,  H , (t  ) must  contain 

-p-1  ^ p-1  p 


the  L -positive  face  which  contains  both  Y and 


Vl- 


Theorem  3 . 5 
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We  are  now  prepared  to  discuss  the  issues  of  the  algorithm  pre- 
sented in  Section  3, 3.2.3. 

Global  Optimality 

The  determination  of  global  optimality  is  the  basic  task  of 

part  (b)  of  Operation  1 of  the  algorithm.  For  this  discussion  let  us 

assume  that  ^ is  a o^-dimensional  operating  point  which  maximizes  the 

restricted  Hamiltonian  H over  the  restricted  constraint  figure  //  . 

P P 

We  have  fixed  time  here.  The  relevant  question  is : 


Do  there  exist  legimate  values  of  for  x^  e 


1 IP 

such  that  H is  maximized  over  //  at  y , where  H is  the 


n-1  dimensional  global  Hamiltonian? 

By  the  basic  geometrical  property  of  linear  programs,  maximizes  H 

over  if  and  only  if  H is  tangent  to  at  Therefore,  the  values 

of  for  X?  e 8 which  we  seek  are  those  which  rotate  H about  H until 
lip  P 

H becomes  tangent  to  ^ at  This  is  illustrated  for  a simple  situa- 

tion in  Figure  3.23. 

In  Section  3.3.4. 1 we  present  a simple  example  of  a constrained 
optimum  that  is  not  globally  optimal. 

Leave-the-boundary  costates 
The  problem  here  is  to 


Determine  the  values  of  X^(t  ) for  all  x^  c L such 

1 p 1 P 
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that  when  maximizing  H , (t  ) the  solution  results  in 

p-1  p 

the  set  of  state  variables  in  leaving  the  boundary 
backward  in  time. 


Now,  statement  (a)  of  Theorem  3.5  says  that  we  essentially  want 
to  rotate  H , (t  ) around  H (t  ) by  varying  A^(t  ) for  x.  e L until 

p-lp  pp'  '^ip  tp 

H , (t  ) touches  ,y  . on  an  1-  -positive  face.  Note  that  this  rotation 

p-1  p p-i  p 

is  performed  while  holding  time  at  t , and  is  to  be  distinguished  from 

P 

the  rotation  of  the  Hamiltonian  which  results  from  the  costates 

evolving  backward  in  time.  See  Figure  3.24. 

If  there  are  no  legitimate  values  of  the  leave-the-boundary 

cc states  which  allow  H , (t  ) to  reach  an  L -ixjsitive  face,  then 

p-lp  p ' 

clearly  it  is  non-optimal  for  the  state  variables  in  to  leave  the 

boundary  at  t . Furthermore,  if  there  is  more  than  one  L -positive 
P P 

face  to  which  H , (t  ) may  be  rotated,  then  the  set  of  all  values  of 

p-1  p 

the  leave-the-boundary  costates  will  be  infinitely  non-unique.  See 
Section  3. 3.4.2  for  an  example  of  this  behavior. 


Subregions 

The  problem  of  determining  subregions  is  a very  difficult  one. 

In  order  to  divide  the  feedback  control  region  into  subregions  with 
respect  to  L^,  it  is  necessary  to  answer  the  following  question: 

What  are  the  collection  of  optimal  control  sets 

52  ,}  and  set  of  breakwalls  asso- 

p-2  p-1 


52  = {52  , . . .,  51 
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ciated  with  the  state  variables  in  leaving  the  boundary 

backward  in  time  at  the  potential  boundary  junction  time  t^ 

for  all  t e [T  , t ^, )? 

P P P+1 


Note  that  the  determination  of  n is  equivalent  to  the  determina- 
tion y = {y  Y 'i  To  get  some  idea  of  what  is  involved 

P~^  P”-i- 

here,  let  us  consider  the  simple  case  in  which  we  assume  that  the 
control  does  not  break  as  time  runs  to  minus  infinity.  In  this  case 
= {n  } , Y = {y  } and  there  are  no  breakwalls  with  which  to  be 

—CO  _oo 

concerned.  The  problem  then  is  to  determine  the  solution  set  Y ^ 
associated  with  the  state  variables  in  leaving  the  boundary  backward 
in  time  at  the  potential  boundary  junction  time  t^  for  all 
t e [t  , t ,).  Let  us  presently  fix  t . Clearly,  Y will  then  con- 

p p p+1  ■'  p ' _oo 

sist  of  points  which  lie  on  L -positive  faces  to  which  H , (t  ) may 

P p-1  P 

be  rotated  (around  In  order  to  determine  exactly  which'points 

of  these  L^-positive  faces  are  in  Y must  be  rotated  slightly 

further  by  perturbing  time  backward  and’sblving  the  constrained  opti- 
mization problem.  In  the  circumstance  that  there  exists  more  than  a 


single  L^-positive  face  to  which  l^*"p^  rotated,  this  pro- 

cedure must  be  performed  for  all  such  faces  in  order  to  determine  the 
total  composition  of  Y We  therefore  must  "probe  ahead"  backward 
in  time  in  order  to  examine  the  ensuing  optimal  solution  Y_^.  Further- 
more, this  must  be  done  for  all  t c [t  , t 

P P P+1 

This  is  undoubtedly  a difficult  problem,  and  does  not  even  take 
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into  consideration  the  general  situation  in  which  the  control  encounters 
breaks.  In  that  case,  we  must  probe  ahead  to  look  at  the  entire  subse- 
quent trajectory  on  (-“,  t^)  for  every  t^  e [x^,  ' Although  no 

technique  is  known  for  solving  this  problem  in  general,  it  is  solved 
for  a simple  three  dimensional  problem  in  Section  3. 3, 4. 3. 

We  have  now  provided  a geometrical  interpretation  of  the  three 
basic  problems  associated  with  the  algorithm.  Unfortunately,  we  do  not 
know  any  way  in  which  to  apply  the  suggested  techniques  in  general . 
However,  in  Section  4.3  we  utilize  the  geometrical  approach  to  prove 
several  simplifications  which  permit  a straightforward  computational 
scheme.  The  special  case  involves  single  destination  networks  with 
all  unity  weightings  in  the  cost  functional.  The  simplifications  are; 

(i)  Every  constrained  solution  is  also  globally  optimal. 

(ii)  The  set  of  leave -the -boundary  costates  is  unique; 
that  is,  there  is  a unique  L^-positive  face  which  can 

be  reached  by  H , (t  ) . In  this  case,  it  is  possible  to 
devise  a computational  technique  for  the  determination 
of  the  values  of  the  leave-the-boundary  costates. 

(iii)  There  is  only  one  subregion  per  region. 

3. 3. 3. 3 Construction  of  Non-Break  Feedback  Control  Regions 

In  this  section  we  specify  the  construction  of  feedback  control 
regions  of  the  type  ./,*  with  associated  control  set  fi  In  s[>eciljt., 

we  assume  that  the  set  of  state  variables  in  L are  leaving  either  a 
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subregion  ) of  some  region  ,'Jt  or  the  breakwall  of  a final  break 

a «p  p p 

feedback  control  region  The  set  of  optimal  controls  with  which 

the  state  variables  in  L leave  persists  without  a switch  until  time 

P 

equals  minus  infinity,  and  is  therefore  denoted 

In  Corollary  3.3  we  showed  that  all  feedback  control  regions  are 
convex  polyhedral  cones.  It  can  be  shown  without  much  difficulty  that 
this  char„.cterization  also  applies  to  both  subregions  and  breakwalls. 
Also,  by  Definition  3.8  we  are  assured  that  the  state  varieibles  in  L 

P 

leave  backward  in  time  from  any  point  in  a given  subregion  or  break- 
wall  with  the  same  set  of  optimal  controls  There  is  consequently 

no  need  to  differentiate  between  the  situation  where  the  state  variables 
in  leave  backward  in  time  from  either  a siibregion  or  a breakwall. 

We  therefore  introduce  the  following  simplifying  notation: 

Notation  3.15:  Both  subregions  and  breakwalls  of  are  referred 

/\ 

to  as  previous  regions,  denoted  '^’p>  when  the  intent  is  to  allow  the 

state  variables  in  L to  leave  from  them  backward  in  time. 

P 

Recall  that  is  the  ^-space  solution  set  corresponding  to  Q 
Since  Y ^ is  the  solution  set  to  a linear  program,  it  is  completely 
described  by  its  extreme  points. 

Definition  3.10:  For  every  extreme  point  of  Y_^,  form  the  ray  in 

0 +p 

R ^ ^ which  originates  at  the  origin  and  passes  through  that  extreme 
point.  If  there  are  u extreme  points,  the  set  of  rays  is  denoted 


This  definition  is  illustrated  in  Figure  3,25. 


a +P 


K i lare  3.  J"")  The  aeL  oT  ray:,  V 


The  rays  of  Definition  3.10  have  a very  special  significance.  Consider 
the  ray  v which  corresponds  to  the  extreme  point  ^ ^ ^ turn, 

corresponds  to  some  optimal  control  ^ c ^ via 


= -X  = -B  u - a. 


= -X 


(3.104) 


Therefore,  for  any  interval  of  time  At  between  switch  times  we  have 
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(3.185) 


where  Ax  represents  the  direction  of  travel  in  forward  time  of  the  state 
under  the  influence  of  the  optimal  control  u in  the  presence  of  inputs 
Therefore,  (3.185)  tells  us  that  ^ represents  the  corresponding 

/s 

optimal  direction  in  x-space  in  which  to  leave  the  previous  region 
By  definition  3.10,  v is  colinear  with  Therefore,  if  we  consider  v 
to  reside  in  jc-space,  it  is  parallel  to  the  optimal  direction  of  travel 
of  the  state.  See  Figure  3.26. 


: 1 (lire  J . 2c 


Interpretation  of  tui  ray  v 
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Note  that  any  optimal  direction  for  leaving  is  expressible 
as  a convex  combination  of  members  of  V ^ and  any  convex  combination 
of  members  of  V is  an  optimal  direction  for  leaving  .-J}  . 

-oo  p 

We  now  denote  by  Co(/  ) the  convex  hull  of  all  the  points  in  the 
set  / . The  following  is  the  baisic  constructive  theorem  for  non-break 
feedback  control  regions. 

Theorem  3.6  Construction  of  non-break  feedback  control  regions) 

Suppose  is  the  set  of  controls  with  which  the  state  variables 

in  L le  ve  .iff  . Then 
P P 

,ff_^  = Co  ( .ff^  u (3.186) 

is  the  non-break  feedback  control  region  with  associated  control  set 
in  the  vein  of  Definition  3.6. 

Proof:  We  must  show  that  items  (i) , (ii)  and  (iii)  of  Definition  3.6 

apply  to  'ff_^  and  The  situation  is  depicted  in  Figure  3.27. 

Items  (i)  and  (ii)  follow  from  the  fact  that  ,ff  is  itself  a feedback 

P 

control  region. 

To  prove  item  (iii)  consider  any  point  x c ,ff  . Translate  each 

- P 

ray  in  V by  placing  its  origin  at  x^  and  call  the  translated  set 
V'  = {v' , v',  ...,  v'}.  Next  form  the  conical  region  Alx)  = 

— ^ U)  — 

Co(x  uv^  )/x.  See  Figure  3.27.  If  x^  e /t(x^),  then  there  exists  a 
direction  which  is  some  convex  combination  of  the  members  of  V which 
takes  x^  optimally  to  x.  Hence,  for  any  x^^  e ^(x)  there  exists  a 


I 


\ 


J 
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u f n which  takes  x optimally  to  x.  Now,  = Co({/l(x)  |x  e 

since  the  smallest  convex  set  containing  {^(j^)|3c  e is  clearly 

.1^  Therefore,  for  any  e .yf  ^ there  exists  some  direction  which 
is  a convex  combination  of  members  of  V which  carries  x,  to  some 

_oo 

point  x £ is  equivalent  to  saying  that  for  any  e ,'/? 

there  exists  a li  £ such  that  x = B IJ,  + ^ carries  to  some  point 

X £ , 

- P 

Now,  let  us  select  some  x^  £ .yf  ^ and  apply  any  control  u B U ^ 
for  any  period  ii  Lime  At  which  keeps  the  resulting  state  x^  in  ,‘/l 
Then  by  ne  eibove  argument  there  exists  some  control  (i^  e which 
takes  J{^  to  some  point  x^  £ See  Figure  3.27.  Now,  the  control  U2 

is  clearly  optimal  since  by  hypothesis  any  u £ is  optimal  to  move 
the  state  off  of  Finally,  u^^  is  optimal  since  the  trajectory  seg- 

ment X.2  “*■  ill  is  part  cf  the  trajectory  x 3 "*■  2L2  2Ll  which  leaves  from 
.yi’p-  We  have  therefore  showed  that  item  (iii)  of  Definition  3.6  is 
satisfied.  ■ Theorem  3.6 


As  is  a convex  polyhedral  cone,  the  convex  hull  construction 
of  Theorem  3.6  results  in  the  convex  polyhedral  cone  This  is  in 
agreement  with  Corollary  3.3.  In  brief,  the  construction  of  non-break 
feedback  regions  proceeds  as  follows: 


(1) 


Begin  with  non-break  control  solution  set  12  ^ corresponding  to 
the  state  variables  in  leaving  the  previous  region  .‘R ^ C R 


backward  in  time. 
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(2)  Find  the  image  Y_^  of  ^ under  the  mapping  (3.162). 

(3)  Form  the  set  of  rays  V_^  from  the  extreme  points  of  Y 

a +p 

(4)  Form  the  conical  region  in  R ^ 

^_oo  = Co(,^p  U V_)/ip. 

The  reader  may  verify  that  all  of  the  constructions  performed  in 
Example  3.1  are  elementary  applications  of  Theorem  3.6.  In  Appendix  B, 
non-break  feedback  regions  are  constructed  for  a three  dimensional 
excimple. 


3. 3. 3. 4 Construction  of  Break  Feedback  Control  Regions 

In  this  section  we  specify  the  construction  of  a feedback  control 

region  associated  with  a control  set  which  applies  on  the  interval 

[T^  Tp) • To  be  more  specific,  we  assume  that  the  state  variables  in 

L are  leaving  backward  in  time  from  either  a subregion  ,^^(L  ) of  some 
P P P 

feedback  control  region  or  the  breakwall  of  a previously  constructed 

break  feedback  region.  In  either  case,  we  invoke  the  convention  of 

Section  3. 3. 3. 3 and  say  that  the  state  variables  in  are  leaving  back- 

ward  in  time  from  the  previous  region  . Our  goal  is  to  construct  the 

P 

break  feedback  region  , associated  with  the  control  set 

p-1  p-1 

The  approach  differs  from  that  for  non-break  feedback  regions 
principally  in  the  definition  of  the  set  of  rays.  We  now  wish  each  ray 


to  represent  an  edge  of  the  breakwall  of  To  find  such  rays,  we 

reason  that  each  edge  of  the  breakwall  serves  as  the  break  surface  of 
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a trajectory  emanating  from  some  edge  of  Additionally,  this  tra- 
jectory will  correspond  to  an  extreme  point  of  although  there  is 

no  way  of  determining  a priori  the  appropriate  one.  We  therefore  define 
the  set  of  rays  for  this  case  as  follows: 


Definition  3.11;  Consider  all  edges  of  the  previous  region  '^p>  Allow 

the  state  variables  in  L to  leave  ))ackv'ard  in  time  from  some  point  on 

P 

each  edge  (wh^re  the  point  may  not  be  the  origin)  once  for  each  extreme 
point  direction  (these  correspond  to  extieuie  points  of 

Let  represent  the  point  of  departure  on  a given  edge.  Then  for 

each  trajectory  find  the  break  point 


x(T  ,)  = x(T  ) + (T  - T ,)(B  u , + a) 

- p-1  - p p p-1 p-1  - 


(3.187) 


where  u is  the  appropriate  extreme  point  of  U For  every  such 

break  point,  form  the  ray  in  R ^ ^ which  origi’-'ates  at  the  origin  and 

and  passes  through  the  break  point.  If  the  total  number  of  break  points 
is  to,  the  set  of  rays  is  denoted 

V 1 = , v , . . . , v } . 

p-1  12  to 

See  Figure  3.28. 


Theorem  3.7  (Construction  of  break  feedback  regions) 

Suppose  i2  , is  the  set  of  controls  with  which  the  state  variables 

p-1 

in  l»-ave  ,/i‘^  backward  in  time.  Then 


• h’, 


P-1 


Co(,>P  IJ  V )/,/i> 

p p-1  p 


(3. 188) 
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is  the  break  feedback  control  region  with  associated  control  set 
in  the  vein  of  Definition  3.6. 


Proof:  We  must  show  that  items  (i) , (ii)  and  (iii)  of  Definition  3.6 

apply  to  "^he  situation  is  depicted  in  Figure  3.26. 

Items  (1)  and  (ii)  follow  from  the  fact  that  .1^  is  itself  a 

P 

feedback  control  region. 

To  prc  e item  (iii)  consider  any  point  >c  F -^p-  Form  the  set  of 
rays  from  ^p_j^  specified  in  Definition  3.10  and  call  them  V.  As 
discussf  * previously,  these  rays  lopiesent  the  directions  of  the  optimal 
trajectories  which  result  from  the  application  of  the  extreme  point 
controls  of  ^p_j^-  Moreover,  the  direction  of  tlie  trajectory  correspond- 
ing to  any  control  in  expressed  as  a convex  combination  of 

of  the  rays  in  V.  Next,  translate  each  ray  in  V by  placing  its  origin 

at  the  point  x and  call  the  translated  set  V . Denote  by  S the  closed 
o +p 

halfspace  of  R ^ ^ which  lies  on  the  same  side  of  the  hyperplane 

which  contains  the  break  wall  as  does  5^.  Consider  the  region  'i(x)  = 

/N 

(Co  (,>?^^  U V ) /x)  n S.  See  Figure  3.29.  If  C 'I  (x)  , then  there  exists 
a direction  which  is  some  convex  combination  of  the  members  of  V which 
takes  optimally  to  x.  Hence,  for  any  x^  e ^(x)  there  exists  a 
u e , which  takes  x,  optimally  to  x.  Now,  ,'/i’  , = Co({^(x)lx  l ./I’  }) 

— p-1  — 1 ‘ — p-1  — (, 


since  the  smallest  convex  set  containing  {^(x)|x  C.  } is 

- p p-1 

liave  therefore  shown  tliat  for  any  x,  c ./j*  , , there  exists  a u t 

—1  p-1  -- 

such  that  X = H u + a carries  x,  to  some  point  x f . 

-1  - P 


We 
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The  reasoning  from  this  point  on  follows  precisely  the  same  lines 
as  that  presented  at  this  point  in  the  proof  of  Theorem  3.6. 

■ Theorem  3 . 7 


As  before,  we  obtain  the  fact  that  ,4^  , as  constructed  in 

p-1 

Theorem  3.7  is  a convex  polyhedral  cone.  In  brief,  the  construction 
of  break  feedback  regions  proceeds  as  follows: 


(1) 


Begin  with  the  break  control  solution  set  ^ corresponding 


to 


the  tate  variables  in  1-  ' 

P 

backward  in  time. 


j the  previous  region  C R 


(2)  Let  the  state  variables  in  L leave  backward  in  time  from  each 

P 

edge  of  with  every  extreme  point  control  in  and  calculate 

the  resulting  break  points  from  (3.187). 

(3)  Form  the  set  of  rays  V ^ from  the  break  points  determined  in  (2). 

^ a +p 

(4)  Form  the  conical  region  in  R ^ 

•Vi-  = ^ Vi’/'^^P* 


The  reader  may  verify  that  the  construction  of  the  break  feedback 
region  labeled  (?)  in  Figure  3.12  is  a simple  application  of 
Theorem  3.7.  In  Appendix  B,  a break  feedback  region  is  constructed  for 
a three  dimensional  example. 


3.3.4  Discussion  of  Fundamental  Properties  of  the  Algorithm 
In  this  section,  we  discuss  several  of  the  properties  of  tlie 
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algorithm  presented  in  Section  3. 3. 2. 2.  We  are  particularly  interested 
in  those  properties  which  are  troublesome  from  the  point  of  view  of 
constructing  a computational  scheme  to  implement  the  algorithm.  The 
discussion  is  focused  on  a simple  example  presented  for  each  property, 
which  serves  not  only  to  illustrate  the  particular  problem  at  hand  but 
also  to  give  the  reader  an  opportunity  to  fix  the  essential  notion  more 
firmly  in  his  mind.  In  these  examples  we  utilize  the  geometrical  point 
of  view  presented  in  Section  3. 3. 3. 2,  so  that  it  is  possible  to  study 
the  properties  by  examining  the  activity  of  the  pointwise  linear  pro- 
gram in  ^-space. 

3. 3.4.1  Non-Giobal  Optimality  of  Certain  Sequences 

Part  (a)  of  Operation  1 of  the  algorithm  consists  of  solving  the 
constrained  optimization  problem  with  a restricted  Hamiltonian.  In 
part  (b)  we  assert  tlie  necessity  to  determine  if  the  solution  to  part 
(a)  can  be  made  globally  optimal  with  the  appropriate  choice  of  costates 
which  satisfy  the  necessary  conditions.  In  this  section,  we  present 
a simple  example  in  which  a constrained  optimum  is  not  a global  optimum, 
thus  justifying  our  concern  for  investigating  global  optimality. 

Example  3.4 

The  general  network.  topolo<}y  to  be  considered  in  this  and  several 
other  examples  is  depicted  in  figure  3.30.  The  link  capacities  are 
indicated  in  brackets  and  for  simplicity  the  inputs  are  all  taken  to  be 


J 
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zero.  The  equations  of  motion  are 


i^(t) 

X2  (t) 

x^(t) 

• 1 

X3(t) 

x^(t) 

x^(t) 


-u 


13 


(t) 

(t) 

(t) 


-u 


*31 

3 

23 

2 

^32 


(t) 

(t) 

(t) 


2 

2 

U33(t) 

+ 

U3i(t) 

1 , . 

1 , . 

U23(t) 

+ 

U32(t) 

u^2(t) 

+ 

u^Vt) 

1 , . 

1 

U32 (t) 

+ 

u^3(t) 

U2^(t) 

+ 

u^2(t) 

2 . 

2 

U3i(t) 

+ 

Ui3(t) 

(3.189) 


For  the  purpose  of  this  example,  we  limit  attention  to  the  state 


variables  x^  and  x^  and  consider  the  cost  function 


[2X3 (t)  + X3 ( t) ] dt. 


(3.190) 


We  are  therefore  considering  a single  destination  networ);  with  non- 
unity weightings  in  the  cost  functional.  The  ^-space  constraint  figure 
is  presented  in  Figure  3.31. 

We  begin  by  letting  x^  leave  the  boundary  bac)tward  in  time  at  t^. 
Then  the  constrained  optimization  problem  in  ^-space  calls  for  the 
maximization  of  the  zero  dimensional  restricted  Hamiltonian 
H^_3(T):z(t)  = constraint  region  .*1  ^ ^ depicted  in 

Figure  3.31.  The  costate  trajectory  for  ir.  sliown  in  Figure  3.32. 

Tlie  solution  to  tlie  constrained  optimization  problem  is 
Y^_j^:y2  = 1.5,  y^  = 0 as  illustrated  in  Figure  3.31.  Now,  at  arbitrary 
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I'iijur.;  3.. 12  k.'os'.ato  i.  ra tor  it_>s  for  t,xani|)>j  3.4 

time  t < let  us  attempt  to  see  if  there  exists  some  legitimate  value 
of  X^(t)  such  that  for  ^2^t)  = 2(t^  - t)  the  Hamiltonian  H(t):z(t)  = 

^2  ^^^^2  ^ 2'  F>^om  Figure  3.31  we  see  that 

the  only  possible  globally  maximizing  Hamiltonian  has  = X^^t). 

However,  from  (3.153)  we  find  that  the  maximum  value  of  X^Ct)  is 
(tj  - t)  , that  is,  one  half  the  value  of  X2(t).  Therefore,  the  candi- 
date operating  point  obtained  from  the  constrained  optimization 

cannot  be  globally  optimal.  We  then  conclude  that  it  is  never  optimal 
for  x^  to  stril^e  the  boundary  last  forward  in  time. 


• Example  3.4 
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3. 3.4. 2 Non-Uniqueness  of  Leave-the-Boundary  Costates 

Refer  to  Definition  3.9  for  the  definition  of  L -positive  face. 

P 

The  significance  of  this  notion  is  established  in  Theorem  3.5,  where 
we  show  that  the  values  of  the  leave-the-boundary  costates  at  t^  are 
actually  those  of  the  appropriate  coefficients  of  l^^p^  rotated 
about  H (t  ) until  it  touches  , on  an  1-  -positive  face.  Here  we 

p p p-1  p ^ 

establish  th^  fact  that  there  may  be  more  than  one  i^-positive  face 
which  may  be  reached  in  this  fashion.  As  we  shall  see,  this  situation 
implies  that  there  may  be  an  infiniLcly  non-unique  cut  of  costates 
corresponding  to  state  variables  which  leave  the  boundary  and  enter 
onto  interior  arcs.  Therefore,  state  variables  which  are  travelling 
on  interior  arcs  may  have  associated  with  them  costates  whose  values 
are  non-unique.  This  is  a most  interesting  property  which  characterizes 
the  linear  state  constrained  optimal  control  problem.  Although  non- 
uniqueness of  costates  may  occur  in  non-linear  state  constrained  prob- 
lems, it  is  usually  limited  to  costates  corresponding  to  state  variables 
travelling  on  boundary  arcs.  See  Bryson,  Denham  and  Dreyfus  [1963]. 

Example  3.5 


Once  again  consider  the  network  topology  of  Figure  3.30, 

12  1 

Let  us  now  limit  attention  to  the  states  x^,  x^  and  and  take 
the  cost  functional  to  be 


1 2 1 
= J (Xj(t)  + x^(t)  + 


) jdt. 


(3.191) 
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The  ^-space  constraint  figure  is  pictured  in  Figure  3.33. 
This  figure  can  be  obtained  by  finding  the  extreme  points  of  U and 


transforming  them  to  ^-space  via  equation  (3.162). 


Suppose  we  let  x^  leave  the  boundary  backward  in  time  at  t^. 


Then  :z(x)  = X^(x)y2  is  maximized  at  the  point 


y^  = y2  = 0 fot  all  X e (-°°,  t^) . This  solution  is  easily  shown  to  be 


globally  optimal  with  = X^(x)  = 0.  The  zero  dimensional 


restricted  Hamiltonian  H^_j^(x)  is  actually  the  point  Y^  ^ in  Figure 
3.33.  This  solution  does  not  break  as  time  runs  to  minus  infinity.  By 


a trivial  application  of  Theorem  3.6  the  underlying  controls  = 0.5, 


u^2  = 1.0,  u^^  = 1.0,  all  other  controls  zero)  apply  everywhere  on  the 


x^  axis. 


2 1 

Suppose  we  know  specify  that  x^  and  x^  leave  the  boundary  simul- 


taneously at  some  arbitrary  boundary  junction  time  E (~°°i  t^)  . 


Now,  since  X^(x)  = -1,  X e ^3^^f-l^  ^f  “ ^f-1" 


L -positive  faces  of  //^  2 this  case,  the  ^-positive  faces)  are 


Fj^,  F^  and  F^  of  Figure  3.33.  Furthermore,  the  two  dimensional  Hamil- 


tonian H^_j^  (t^_j^)  :z(t^_j^)  = ^3(t^_^)y2  + 


be  rotated  about  the  zero  dimensional  Hamiltonian  H^  i^*"f-l^  touch 


in  all  faces  F^,  F^  and  F^.  In  fact,  it  may  lie  on  F^  anywhere 


between  F.  and  F^.  Therefore,  the  leave-the-boi  ndary  costates  X,  and 


X2  at  t^_j^  may  achieve  values  anywhere  between 
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and 


V^Vi'  •“ 

(*^Vx>  ■ “ 


Hence,  we  have  an  infinitely  non-unique  set  of  leave-the-boundary  co- 

2 1 

states  at  This  non-uniqueness  persists  even  after  and 

leave  the  boundary  and  enter  onto  interior  arcs,  thus  verifying  our 
contention  in  the  introduction  to  this  section.  •Example  3.5 


In  general,  the  backward  constructive  algoritlm  calls  for  the 

entire  set  of  costate  values  if  they  are  indeed  non-unique.  However, 

it  is  clear  that  Hamiltonian  hyperplanes  which  lie  between  faces  of  ,</ 

will  provide  us  with  no  more  information  in  terms  of  new  operating 

points  than  will  those  which  actually  lie  on  tlie  faces.  If  the  state 

/\ 

variables  in  L leave  some  region  ;'/?  , we  therefore  must  find  all  of 
P P 

the  highest  dimensional  L -positive  faces  of  , upon  which  H , (t  ) 

P p-1  p-1  P 

may  be  made  to  lie.  We  then  utilize  the  finite  costate  set  correspond- 
ing to  the  coefficients  of  H , (t  ) which  bring  it  to  lie  on  these 

p-1  P 

Lp-positive  faces.  In  Example  3.5,  this  costate  set  corresponds  to  the 
faces  and  F^. 

The  reduction  of  the  required  costates  to  a finite  set  is  a 
welcome  result  but  we  still  have  the  problem  of  finding  the  appropriate 
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i-^-positive  faces  of  Although  this  is  a simple  problem  for 

three  dimensions  or  less,  we  have  no  method  for  performing  this  task 
for  a general  dimensional  multi-destination  network.  The  solution  to 
this  problem  is  contingent  upon  discovering  a method  for  explicitly 
determining  the  constraints  which  determine  , in  the  vicinity  of 

p-1 

tlie  operating  point  In  Section  4.4  we  show  how  this  may  be 

achieved  whe.  the  L^-positive  face  is  unique  as  in  the  case  of  problems 
involving  single  destination  networks. 


3, 3.4. 3 Subregions 


Recall  that  a subregion  ,'J}  (L  ) of  a region  is  that  set  of 

P P P 

points  in  which  when  taken  as  the  point  of  departure  of  the  state 
variables  in  L result  in  a common  control  set  and  common  set  of  break- 


P 

walls  (Definition  3.8) . We  now  present  an  example  in  which  there  are 
two  subregions  in  a particular  feedback  control  region. 


Example  3.6 

The  network  is  picture  in  Figure  3.34.  Once  again,  for  simpli- 
city we  are  considering  the  no  inputs  case.  This  is  a single  destina- 
tion network  with  all  messages  intended  for  node  4;  therefore,  we  may 
eliminate  the  destination  superscript  on  the  state  and  control  vari- 
ables. The  dynamical  equations  are 

Xi(t)  = ^2^(1)  + 

x^Ct)  = (3.192) 

x^(t)  = — u^l^(t) 


i 


t 

i 
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Fiqure  3,34  Network  'I'opoloqy  for  Kxample  3.6 


and  we  consider  the  cost  functional 


J 


[2Xj^(t)  + x^Ct)  + 2x2(t)]dt. 


(3,193) 


The  ^-constraint  figure  is  depicted  in  Figure  3.35(a). 

We  begin  by  letting  x^  leave  the  boundary  backward  in  time  at 
t^.  The  constrained  optimization  problem  calls  for  the  maximization  of 
the  zero  dimensional  Hamiltonian  (t)  :z (T)  = X2(T)y2  over  the  con- 
straint figure  depicted  in  Figure  3.35(a).  The  costate  trajectory 

for  is  shown  in  Figure  3.35(b).  The  solution  to  the  constrained 
optimization  problem  is  ~ ^ ^ 2 ~ ^3  “ Moreover,  it  is 

easy  to  see  by  examining  Figure  3.35(a)  that  this  solution  is  globally 
optimal  for  the  costate  values  Aj^(i)  = A^(t)  = 0,  Also, 


persists 


as  time  runs  to  minus  infinity.  The  trajectory  is  illustrated  in 
Figure  3.36.  Utilizing  the  controls  underlying  this  solution,  we 
are  able  to  assign  the  feedback  control  on  the  X2~axis  as  specified  in 
Figure  3.37.  We  denote  this  region 

Next  we  stipulate  that  x^  leave  the  boundary  at  some  arbitrary 
boundary  junction  time  t^  The  one  dimensional  Hcimiltonian  that 
touches  the  constraint  figure  2 the  positive  orthant  of  y^  is 
simply  H^_2  (t^_^)  :z  (t^_j^)  = ^2^*^f-1^^2‘  Therefore,  = 0.  As 

we  proceed  backward  in  time  ftom  t^  the  one  dimensional  constrained 
Hamiltonian  H^  2(T):z(T)  = ^2  ^^^^2  ^ maximized  over  .'/j_2 

at  the  point  \ ~ ^2  ~ A*®'  y3  = 1.0.  This  solution  is  glo- 

bally optimal  for  X^(T)  = 0 and  does  not  experience  a break  as  time 
runs  to  minus  infinity.  We  may  then  apply  Theorem  3.6  to  construct, 
the  two  dimensional  non-break  feedback  control  region  labeled 
Figure  3.37. 

We  now  want  to  allow  Xj^  to  leave  the  boundary  backward  in  time 
at  some  time  2 ^ ^f  1'  allow  the  state  to  leave  from  '^^_2‘ 

This  is  achieved  by  rotating  the  two  dimensional  Hamiltonian  H^  3^'-f  2^ 

z(tf_2>  = ^2^*"f-2^^2  ^ ^3^^f-2^^3  '^f-2^^f-2^  until 

it  touches  //  on  a face  in  the  positive  orthant  of  y^^.  By  examining 
Figure  3.35(b)  and  (c)  we  distinguish  the  following  cases: 

(i)  If  ^ ^2^*"f-2^'  3^''f-2^  rotated  to  touch 

the  edge  labeled  e'  in  Figure  3.35(b).  This  edge  becomes  the 


I n 

Fnure  3.37  The  Subregions  *^t- 

of  the  reedlv*t:k  Control  i<eqion 


new  set  of  operating  points  ^£3* 

(ii)  If  ^,(t_  _)  = X (t,  „) , then  H,  ^(t,  „)  is  rotated  to  touch 

the  face  labeled  in  Figure  3.35(c),  Subsequent  rotation 

of  2 causes  the  edge  e"  of  Figure  3.35(c)  to  become  the 
new  set  of  operating  points 

(iii)  If  ^ ^2^^f  2^'  ^f-3^'"f-2^  rotated  to  touch 

the  e ;e  labeled  e"  in  Figure  3.35(b).  This  edge  becomes 

the  new  set  of  operating  points  3-  | 

If  we  deiiote  the  time  at  which  equals  X^  by  T^,  then  from  Figure  | 

3.36  we  easily  determine  that  ~ 2x^(1^).  Therefore,  we  di^'ide 

the  region  ,:>P^  into  two  subregions:  2^^1^  that  portion  of 

beneath  the  line  X2=2x^,  not  including  X2=2x^;  is  that  j 

portion  of  2 above  and  including  the  line  X2=2x^.  When  the  state  ^ 

leaves  2^^1^  operating  points  is  ^ = e'.  On  the 

other  hand,  when  the  state  leaves  2^^1^  operating 

points  is  Y£_3  = e",  •Example  3.6 

\ 

3, 3.4.4  Return  of  States  to  Boundary  Backward  in  Time 

In  Operation  1 of  the  algorithm  we  worlc  under  the  assumption  ■ 

that  no  state  variable  which  is  off  the  boundary  must  subsequently  \ 

return  to  the  boundary  bac)iward  in  time.  This  is  equivalent  to  assuming  | 

1 

that  in  forward  time  optimality  will  never  require  that  a state  variable  j 

increase  its  current  value.  In  Section  4 . 3 we  show  that  this  assumption 
is  in  fact  true  tor  the  case  of  single  destination  networks  wit)i  all 
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unity  weightings  in  the  cost  functional.  However,  in  general  this 
assumption  is  not  true  as  illustrated  by  the  following  example. 

Example  3.7 

We  once  again  refer  to  the  network  topology  of  Excimple  3.4  and 
concern  ourselves  with  the  state  variables  x^  and  x^  and  the  cost 
functional  (3.190)  of  Example  3.4.  As  in  that  case,  the  y-constraint 
figure  appears  as  in  Figure  3.31,  which  we  present  again  in  Figure  3.36 
for  illustration  on  the  current  discussion. 

In  Example  3.4,  we  attempted  to  allow  x^  to  leave  the  boundary 
backward  in  time  at  t^  and  discovered  that  this  trajectory  cannot  be 
globally  optimal.  We  now  try  letting  x^  leave  the  boundary  backward  in 
time  at  t^.  Then  the  constrained  optimization  problem  calls  for  the 
maximization  of  the  zero  dimensional  restricted  Hamiltonian  ^(l): 
z(t)  = A2^T)y2  over  the  constraint  figure  depicted  in  Figure 

3.38.  The  costate  trajectory  is  shown  in  Figure  3,39.  The  solution 
to  the  constrained  optimization  problem  is  y^  = 0.  This 

solution  can  be  shown  to  be  globally  optimal  for  the  costate  values 

= A^^t)#  T e ^ Figure  3,38.  Also,  persists 

as  time  runs  to  minus  infinity.  Over  this  interval  x^  travels  forward 
in  time  with  a slope  of  -1.5  as  illustrated  in  Figure  3.39. 

Now,  suppose  at  an  arbitrary  boundary  junction  time  t^  we 
choose  to  allow  x^  to  leave  the  boundary  backward  in  time.  Then  the 
costate  = ^2**^f-l^  * *"f  ~ '^f-l 


causes  the  enlarged  Hamiltonian 


Figure  J.38  Geometrical  Interpretation  of  Situation 
in  Which  State  Variable  Returns  to 
Boundary  Backwurd  in  Time 


-200- 


to  lie  against  the  ^-positive  face  of  depicted 

in  Figure  3.38.  Once  leaves  the  boundary  backward  in  time  its 
costate  travels  with  a forward  slope  of  -2  as  indicated  in  Figure 
3.39.  The  Hamiltonian  ^(T)-.z(T)  = ^ ^3*^*^3  therefore 

achieves  its  maximum  over  //^  ^ at  2‘^3  ~ ~^-^r  - 2.0.  Hence, 

the  optimal  slope  of  x^  forward  in  time  is  now  +0.5.  The  trajectory 
is  pictured  Figure  3.39  where  we  see  that  x^  returns  to  the  boundary 
at  the  time  which  is  3(t^  - t^_^)  time  units  before  t^  The 

return  rf  x^  to  the  boundary  marks  i "dead  end"  of  the  backward  trajec- 
tory since  it  is  not  possible  to  allow  x^  to  re-enter  the  interior 
backward  in  time.  This  is  because  in  order  to  do  so  it  is  necessary 

to  have  _)  in  order  to  have  ^(T,  _)  lie  on  the 

3 t-2  2 f-2  f-2  t-2 

appropriate  face  of  //^ 

However,  nothing  is  lost  since  the  trajectory  of  Figure  3.39  may 
be  used  to  construct  the  feedback  solution  for  ail  initial  conditions 
on  x^  and  x^.  We  simply  treat  ^ a break  point  and  find  that  the 
entire  space  is  filled  up  upon  the  construction  of  the  break  feedback 
region.  See  Figure  3.40.  # Example  I."" 

We  have  presented  a simple  example  for  which  a state  variable 
returns  to  the  boundary  backward  in  time  and  demonstrated  how  the 
feedback  space  may  be  filled  up.  In  this  particular  case,  the  point 
at  which  the  state  variable  returns  to  the  boundary  represents  a "dead 
end".  However,  it  is  not  clear  that  this  is  true  in  general.  In  order 


r 
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to  fill  up  the  space,  it  may  be  necessary  (and  therefore  optimal)  to 
allow  state  variables  which  have  returned  to  the  boundary  backward  in 
time  to  re-enter  the  interior.  Once  again,  we  require  explicit  know- 
ledge of  the  ^-constraint  figure  in  order  to  accommodate  this  problem 
as  part  of  a computational  algorithm. 

Before  leaving  the  discussion  of  this  property,  we  point  out 
that  there  e Ists  some  connection  between  the  problem  of  non-global 
optimality  and  that  of  state  variables  returning  to  the  boundary  back- 
ward in  time.  We  can  infer  thi'  '■  xamining  the  conclusions  of 
Examples  3.4  and  3.7,  which  were  performed  for  the  same  network  topology 
and  cost  functional.  In  Exeimple  3.4,  we  showed  that  it  is  not  globally 
optimal  for  x^  to  strike  the  boundary  last.  This  is  consistent  with 
the  feedback  solution  of  Example  3.7,  which  calls  for  x^  to  leave  the 
boundary  forward  in  time  whenever  x^  has  a positive  value.  It  is  not 
clear  how  this  notion  may  be  expanded  to  characterize  general  network 
problems. 

3. 4 Summary 

In  this  chapter  we  have  developed  an  algorithm  which  constructs 
the  feedback  soluti'.n  to  the  dynamic  routing  problem  when  the  inputs 
to  the  network  are  constant  in  time.  The  building  blocks  of  the  feed- 
back space  are  convex  polyhedral  cones  and  we  have  proposed  constructing 
these  cones  by  a sequence  of  optimal  trajectories  run  backward  in  time. 


These  trajectories  are  fashioned  to  satisfy  the  necessary  conditions. 
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vrtiich  are  also  sufficient. 

Through  the  process  of  developing  a rigorous  algorithm  we  have 
discovered  the  fundamental  characteristics  of  the  feedback  solution. 
Aside  from  its  deceivingly  neat  organization  into  convex  polyhedral 
conical  regions,  the  feedback  space  is  a complex  and  oftentimes  con- 
fusing creature.  This  confirms  the  viewpoint  commonly  shared  among 
optimal  control  theorists  that  feedback  solutions  are  very  difficult 
to  find,  especially  for  problems  involving  constraints  on  the  state 
variables. 

The  complex  algorithm  for  the  construction  of  the  feedback  space 
developed  in  this  chapter  for  the  general  network  problem  is  difficult 
to  implement  in  a computational  scheme.  However,  it  serves  as  a con- 
ceptual setting  for  proving  the  simplifications  which  result  for 
problems  involving  single  destination  networks  with  unity  weightings 


in  the  cost  functional. 


I 


I 


Chapter  4 

FEEDBACK  ROUTING  WITH  CONSTANT  INPUTS 
AND  UNITY  Vre:iGHTINGS 


4.1  Introduction 

In  Chapter  3 we  presented  an  algorithm  for  the  construction  of 
the  feedback  solution  to  the  general  network  dynamic  routing  problem 
with  constant  inputs.  However,  we  found  that  four  basic  properties 
associated  with  the  algorithm  complicate  the  formulation  of  a computa- 
tional scheme  to  implement  the  algorithm.  In  brief,  these  are: 

1)  Non-global  optimality  of  certain  sequences  of  state  variables 
leaving  the  boundary  backward  in  time 

2)  Non-uniqueness  of  leave- the-boundary  costates 

3)  Subregions 

4)  Return  of  state  variables  to  boundary  backward  in  time. 

In  this  chapter,  we  investigate  these  properties  as  they  apply 
to  problems  involving  single  destination  networks  with  inputs  constant 
in  time  and  all  unity  weightings  in  the  cost  functional.  Throughout 
Chapter  3 we  have  promised  that  for  this  class  of  problems  various 
simplifications  result.  Here  we  state  and  prove  these  simplifications 
using  the  geometrical  point  of  view  presented  in  Section  3. 3.3.2. 
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The  primary  results  are  (in  the  order  they  are  proved  in  Section  4.3); 

Theorem  4.1  The  leave- the-boundary  costate  values  are 
unique . 

Theorem  4 , 2 There  is  one  subregion  per  region. 

Theorem  4.3  All  solutions  to  the  constrained  optimization 
problem  are  also  globally  optimal. 

Theorem  4.4  There  always  exists  an  optimal  control  which 

does  not  call  for  the  return  of  state  variables 
to  the  boundary  backward  in  time. 

One  additional  result  which  even  further  simplifies  the  situation  is; 

Lemma  4.6  No  breaks  in  the  optimal  control  between 
boundary  junction  times. 

As  a consequence  of  this  last  property,  we  need  only  solve  the  point- 
wise  constrained  linear  program  once  between  appointed  boundary  junction 
times.  Moreover,  the  only  kind  of  feedback  regions  to  be  constructed 
are  those  of  the  non-break  type  detailed  in  Section  3. 3. 3. 3. 

With  these  simplifications  in  hand,  we  proceed  to  construct  the 
details  of  a computational  scheme  which  realizes  the  backward  construc- 
tive algorithm  for  this  class  of  problems.  Of  particular  interest  is 
the  use  of  parametric  linear  programming  techniques  to  determine  the 
L^-positive  face  which  provides  the  values  of  the  leave-the-boundary 
costates.  A computer  program  is  developed  to  implement  the  scheme  for 


a five-node  network.  However,  the  portion  of  the  program  which  employs 
Chernikova's  algorithm  to  calculate  the  entire  set  of  solutions  to  the 
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pointwise  linear  program  is  highly  inefficient.  This  problem  causes 
the  calculation  of  the  entire  feedback  solution  to  be  impractical . 
Finally,  we  discuss  some  of  the  issues  involved  in  extending  the  geo- 
metrical approach  to  general  network  problems. 


4.2  Preliminaries 


As  we  shall  presently  become  involved  in  detailed  geometrical 
analysis  in  the  ^-space,  we  briefly  review  a few  of  the  essential 
notions.  First  lecall  that  the  definition  of  the  ^-constraint  figure 
is : 


•V  = {^  e I il  ~ H ^ u e U } 


where  U={u:Du<c  and  u > O}. 


(4.1) 


Next  recall  that  the  constrained  optimization  problem  of  Section 

3.3.2. 2 (Operation  1,  part  (a))  calls  for  the  maximization  of  the 

Hamiltonian  H (x)  over  the  restricted  constraint  figure  , where  we 
P P 

define  //  as  follows: 

P 

.0  . , 

V = (y  e R ^(y  6 V and  y?  = 0 V x?  e 6 } . (4.2) 

■ 'p  i-  1 p 


We  remind  the  reader  that  B is  the  set  of  state  variables  on 

P 

the  boundary  for  T t;  (T  , r .,).  Also,  I is  the  set  of  state  vari- 

p p+1  p 

ables  off  the  boundary  for  T C [X  , X .,) , where  card  (I  ) = 0 . Now, 

P P+1  P P 

the  solution  set  to  the  constrained  optimization  problem  on 

T £ fx  , I , ) is  denoted  by  Y . A member  of  Y is  denoted  by  y , 
p p+1  P P +1' 

is  referred  to  as  a Op-dimensionai  operating  (xaint. 


and 
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The  following  proposition  regarding  and  ^ are  easy  to  prove, 

P 


so  that  the  details  are  eliminated  here. 


Proposition  4.1;  is  a closed  and  bounded  convex  polyhedron.  The 

scune  is  true  of  y . 

P 

The  notions  of  extreme  point  and  edge  of  a convex  polyhedron  are 
central  in  linear  programming.  Rigorous  geometrical  definitions  of 
these  notions  may  be  found  in  Dantzig  [1963] . Briefly,  they  are: 


Definition  4.1;  A point  of  a convex  set  is  an  extreme  point  of  that 
set  if  no  straight  line  segment  of  finite  length  completely  in  the  set 
contains  the  point. 

Definition  4.2;  The  edge  of  a convex  polyhedron  is  the  straight  line 
segment  joining  two  adjacent  extreme  points. 


Now,  from  linear  programming  considerations  and  the  details  of 


our  algorithm  we  know  that  is  the  convex  polyhedron  which  is  the 

face  of  tangency  between  the  restricted  Hamiltonian  hyperplane  Hp(t) 

and  the  restricted  constraint  figure  on  x e [x  » T Therefore, 

P P P+1 

extreme  points  of  the  solution  set  Y^  are  also  extreme  points  of  .y^. 
The  following  proposition  is  easy  to  prove. 


Proposition  4.2;  The  solution  set  to  the  constrained  optimization 
problem,  Y , lies  on  the  boundary  of  .y.  However,  extreme  points  of 
are  not  necessarily  extreme  points  of  ,y. 

* ' T\ 
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Now,  we  present  a proposition  which  provides  the  fundaraental 
geometrical  relationship  between  the  constraint  figures  and  U.  Once 
again,  the  proof  is  simple  and  is  eliminated  for  the  purpose  of  brevity. 

Proposition  4.3:  Each  extreme  point  of  is  the  image  (through  the 

mapping  Y.  " “5.  ii  ” exactly  one  extreme  point  of  U.  However, 

extreme  points  of  U may  map  into  points  that  are  not  extremum  of 
(such  as  boundary  and  interior  points) . Likewise,  each  edge  of  is 
the  image  of  exactly  one  edge  of  U.  However,  edges  of  U may  map  into 
points  which  are  not  edges  of  ;V. 

Now,  in  order  to  describe  the  convex  polyhedron  //  we  need  only 
concern  ourselves  with  its  extreme  points,  or  equivalently,  with  its 
edges.  However,  we  do  not  have  an  explicit  representation  of  in 
terms  of  linear  inequalities,  so  that  we  must  approach  this  problem 
somewhat  indirectly.  The  approach  shall  be  to  talk  about  edges  of  U, 
for  which  we  do  have  a representation,  and  then  to  investigate  the 
manner  in  which  the  mapping  ^ u ” a acts  upon  these  edges.  In 

particular,  we  shall  be  exploiting  the  special  structure  of  the 
matrices  B,  C and  D (where  £,  D and  C appear  as  in  (4.1)).  We  present 
here  the  argument  as  far  as  it  applies  to  single  destination  networks. 
This  argument  is  extended  to  cover  multi-destination  network  models  in 
Section  4.5,  where  the  subsequent  application  of  the  argument  is  not 
quite  as  clear  as  it  is  here. 

We  begin  by  examining  the  nature  of  the  edges  of  U.  The  point 
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of  view  that  we  take  to  achieve  this  task  is  the  algebraic  interpreta- 
tion of  linear  programs.  This  interpretation  is  basic  in  the  formula- 
tion of  algorithms  to  solve  linear  programs,  and  as  such  can  be  found 
in  any  good  text  on  the  subject  (such  as  Dantzig  [1963]).  The  basic 
notions  are : 

(i)  the  canonical  form  of  a system  of  linear  equations, 

(ii)  a corresponding  basic  feasible  solution, 

and  (iii)  a pivot  from  one  basic  feasible  solution  to  another, 

performed  by  exchanging  a basic  variable  for  a non-basic 
variable  while  maintaining  canonical  form. 

Now,  extreme  points  of  a linear  constraint  set  are  characterized 
algebraically  as  corresponding  to  basic  feasible  solutions  of  the 
representative  system  of  linear  equations  in  canonical  form.  Also,  a 
pivot  from  one  basic  feasible  solution  to  another  corresponds  geometri- 
cally to  moving  along  an  edge  from  the  extreme  point  represented  by  the 
first  basic  feasible  solution  to  the  extreme  point  represented  by  the 
second  basic  feasible  solution.  This  is  the  important  property  which 
we  shall  exploit  in  our  attempt  to  describe  edges. 

We  now  examine  the  set  of  equations  which  describes  U for  single 
destination  networks.  The  capacity  constraint  over  a link  (i,k)  has 
now  tile  simple  form 

0 < u,,  < C.,  (4.3) 

— ik  — ik 


since  for  a single  destination  there  is  exactly  one  control  variable 


-210- 


I 


I 

I 

1-, 


per  link.  We  notice  immediately  from  (4,3)  that  U is  a rectangular 
hyper-parallelepiped  so  that  the  algebraic  description  of  the  edges  is 
simple  to  obtain.  However,  we  shall  now  characterize  an  edge  alge- 
braically in  a rigorous  fashion  in  order  to  establish  the  basic 
approach  for  the  more  complicated  multi-destination  situation  discussed 
in  Section  4.5. 

We  begi'  by  introducing  the  non-negative  slack  variable  so 
that  the  inequality  u^^^  ^ becomes 


^k  ^ "ik  = ""ik 


z.,  > 0. 
ik  — 


(4.4) 


The  complete  set  of  equations  of  type  (4.4)  corresponding  to  £ ^ £ 

therefore  has  the  form: 


r r 


where  we  have  eliminated  all  subscripts  except  for  ik  to  avoid  nota- 
tional  havoc.  Recall  that  r is  the  dimension  of  the  control  vector  and 


therefore  is  the  dimension  of  the  link  collection  L.  Now,  (4.5)  is 
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immediately  in  canonical  form  with  respect  to  any  set  of  r variables 

chosen  (either  u’s  or  z's)  to  be  basic.  For  instance,  if  we  consider 

the  row  corresponding  to  then  we  pick  either  or  to  be 

basic  and  set  its  value  to  and  the  value  of  the  remaining  variable 

to  zero.  Note  that  such  basic  solutions  always  have  u^j^  ^ 0 (u^j^  = 0 

or  u,,  = C.,  ) so  that  the  lower  bound  in  (4.3)  is  never  violated  when 
ik  ik 

talking  about  basic  solutions  to  (4.5). 

Hence,  every  extreme  point  of  U is  represented  by  a point  whose 

ik  coordinate  has  value  of  either  zero  or  C.,  . In  terms  of  the  net- 

ik 

work,  this  means  that  every  link  is  operating  either  at  zero  or  full 
capacity.  Now,  in  order  to  perform  a pivot  on  (4.5) , we  simply  pick 
one  variable  which  is  currently  basic  at  value  (either  u^j^  or 

z^j^)  and  replace  it  in  the  basis  by  the  remaining  variable  (either 
z^j^  or  u^j^  respectively).  Hence,  movement  along  an  edge  of  U is  equiva- 
lent  to  picking  a coordinate  (say  ik)  of  the  current  extreme  point  and 


either  increasing  its  value  to  (if  it  is  currently  zero)  or  de- 

creasing its  value  to  zero  (if  it  is  currently  at  C^j^)  . The  total 
number  of  edges  emanating  from  any  extreme  point  is  therefore  r. 

We  nov/  are  in  a position  to  discuss  the  nature  of  edges  of  .y , 
since  by  Proposition  4.3  edges  of  correspond  to  edges  of  (i.  First 
we  distinguish  between  the  two  basic  types  of  control  varicibles. 


Definition  4.3:  Suppose  the  single  destination  of  the  network  is 


node  d.  Then  a control  variable  of  the  type  u. ^ is  called  a direct 

id  
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control.  A control  variable  of  the  type  k d,  is  called  an 

intermediate  control.  See  Figure  4.1. 


Figure  4.1  DicocL  and  Intermediate  Coiitrol  Varicibles 


Now,  a direct  control  enters  into  exactly  one  component  of  the 
dynamic  equation  ^ a since  there  is  no  state  variable  associated 

with  the  destination  node.  The  contribution  is 


’‘i 


= -u.  , + 
id 


+ a , 
1 


(4.6) 


where  we  have  eliminated  the  single  destination  superscript  on  the 
state  and  control  variables  and  input.  Also,  an  intermediate  control 
enters  into  exactly  two  components  of  the  dynamic  equation  as  follows 


X.  = -u.,  + 

1 ik 


\ = "“ik 


1 

k* 


+ ...  + a, 


(4.7) 
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Suppose  now  that  the  current  operating  point  ^ ^ is  an  extreme 

point  of  We  wish  to  characterize  the  types  of  edges  of  ^ which 

may  emanate  from  From  (4.6)  and  (4.7)  we  distinguish  the  following 

two  types  of  edges  of  ^ : 


Type  1 - Exactly  one  coordinate  of  ^ changes  from  its  value 
at  y^.  This  edge  corresponds  to  an  edge  of  U for  which  £ 
direct  control  and  its  slack  change  places  in  the  basis. 
From  equations  (3.162)  and  (4.6)  we  see  that  the  two  possi- 
bilities are: 


_ 

0 

id 

pivot  j id 

c 

) z 

id 

id 

( 

id 

^id 

(u 

pivot  1 id 

id 

0 

=>  ^^i  = ^id 


(4.8) 


Ay.  = -C. , 
1 id 


(4.9) 


where  Ay^  indicates  the  change  in  that  ith  coordinate  which 
occurs  when  moving  from  one  end  to  another  of  an  edge.  Notice 
that  Ay,  is  either  plus  or  minus  C,  . 


Type  2 - Exactly  two  coordinates  of  ^ change  from  their  value 
at  This  edge  corresponds  to  an  edge  of  for  which  an 

intermediate  control  and  its  slack  change  places  in  the  basis. 
From  equations  (3.162)  and  (4.7)  we  see  that  the  two  possibili- 


ties are: 
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“ik 

= 0 

pivot 

= ^ik 

< 

- ■'ik 

^ik 

" ^ik 

k. 

= 0 

1 

^^^k  = -^ik 

'^ik 

= ^ik 

pivot 

= 0 

=>j 

[""i  = -"ik 

^ik 

= 0 

= "ik 

u 

II 

>1 

<1 

(4.10) 


(4.11) 


where  Ay^  and  indicate  the  changes  in  the  ith  and  kth 

coordinate  respectively  which  occur  when  moving  from  one  end 

of  an  edge  to  another.  Notice  that  Ay.  and  Ay,  are  both  either 

1 k 

plus  or  minus  and  that  Ay^  = “Ay^^. 


Notation  4.1;  An  edge  of  Type  1 in  which  the  change  occurs  in  the 
coordinate  y^  is  represented  as 

t 

^i 

and  an  edge  of  Type  2 in  which  the  change  occurs  in  the  coordinates 
y^  and  yj^  is  represented  as 


'k' 


We  now  present  a brief  refresher  on  the  geometrical  point  of  view 
applied  to  the  analysis  of  various  properties  associated  with  the  back- 
ward construction  algorithm.  Assume  that  a current  a -dimensional 

P 

operating  point  Yp  E the  restricted  Hamiltonian  hyper- 


plane is 
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H (T)  :z(T)  = 


^ / T \ 


1 P 


(4.12) 


T E [Z  , T ) . 

P P+1 

We  now  choose  to  allow  the  state  variables  in  L to  leave  the  boundary 

P 

backward  in  time  at  the  appointed  boundary  junction  time  t^  e ^^p'^p+1^ " 
Note  that  may  represent  a control  switch  time  or  the  boundary  junc- 
tion time  t , before  t in  backward  time.  Then  the  values  of  the 
P+1  P 

leave-the-boundary  costates  X^  (t  ) for  x?  c L are  those  for  which  the 

1 p 1 P 

enlarged  Hamiltonian  hyperplane 


H ,(t  ):z(t  ) = .1]  Xl(t  )yj  + X^ (t  )y^ 

P-1  P P ^ P " xhL  ^ P " 

1 p 1 p 


(4.13) 


contains  an  L -positive  face  of  ,'V 

P P-1 


Notation  4.2:  Let  be  a current  set  of  operating  points  and  be 

the  set  of  state  variables  to  leave  the  boundary  backward  in  time  at 
time  t e (t  , T l . Then  an  L -positive  face  of  u , which  is 

p p p+1  p ^ '•'p-1  

contained  in  H , (t  ) for  some  ' ? (t  ) , e L , is  denoted  F(Y  ,t  ,L  ). 
p-1  p 1 p 1 p P P P 

The  collection  of  all  such  faces  is  denoted  , t , L ). 

P P P 

This  completes  our  discussion  of  the  geometrical  preliminaries 
required  in  the  proofs  of  the  next  section.  Additionally,  we  shall 
have  occasion  to  utilize  some  basic  terminlogy  and  elementary  results 
from  graph  theory.  The  relevant  definitions  and  a simple  result  are 
presented  here. 


.J 
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Definition  4.4:  A graph  is  a set  of  nodes  ...»  y^}  and  a set  of 

ordered  pairs  y^  called  edges . We  call  y^  and  y^  extremities  of 

the  edge. 

Note  that  we  have  actually  referred  to  the  coordinates  of  ^ as  the 
nodes.  This  shall  be  our  usage.  Edges  of  the  form  y^  y^  actually 
correspond  In  our  discussion  to  edges  of  emanating  from 

Definition  4.5:  A subgraph  of  a given  graph  is  a graph  which  contains 

a subset  of  the  nodes  and  edges  oi  the  given  graph. 

Definition  4.6:  A chain  is  a sequence  of  edges  such  that  one  of  the 

extremities  of  each  edge  belongs  to  the  preceding  edge  in  the  sequence 
and  the  other  extremity  to  the  following  edge. 

Definition  4.7:  A graph  is  connected  if  two  arbitrary  distinct  nodes 

are  always  connected  by  a chain. 

Definition  4.9:  A connected  graph  without  loops  is  a tree . 

Proposition  4.4;  A tree  of  s nodes  contains  s-1  edges. 

4 . 3 Special  Properties  of  the  Algoritlim 

We  begin  with  the  central  theorem  of  this  section.  The  proof  of 
this  theorem  employs  the  geometrical  point  of  view,  and  contains  tlie 
basic  notions  which  are  utilized  in  subsequent  proofs. 
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Theorem  4.1 

For  problems  involving  single  destination  networks,  the  values  of 

the  leave- the-boundary  costates  are  unique  at  a given  boundary  junction 

time  t . 

P 


Proof ; We  emphasize  that  in  this  proof  t^  is  a fixed  potential  boundary 

junction  time,  and  begin  by  considering  the  case  in  which  we  wish  to 

allow  all  the  states  remaining  on  the  boundary  to  leave  the  boundary  at 

t , that  is,  L = B . It  will  be  easy  to  generalize  the  result  to 
p P P 

cases  for  which  L C 8 and  L B . 

P P P P 

We  now  divide  the  proof  into  two  principal  sections: 


(a)  contains  an  extreme  point  of 

(b)  Y does  not  contain  an  extreme  point  of 

P 


(a)  y contains  an  extreme  point  of  fy. 


The  proof  will  proceed  by  contradiction.  Suppose  the  values  of 

the  leave- the -boundary  costates  are  not  unique.  Then  according  to  the 

geometrical  interpretation  , t , 8 ) contains  at  least  two  dis- 

P P P 

tinct  members.  Call  these  F'(Y  , t , 8 ) and  F" (Y  , t , 8 ) . Each 

P P P P P P 

of  these  is  a face  of  fy  since  L =8  and  :U  .=  A . Now,  the 

P P p-1 

enlarged  Hamiltonian  hyperplane  is  actually  the  n-1  dimensional  global 

Hamiltonian  hyperplane,  so  that  we  refer  to  it  as  lUt^).  In  accordance 

with  the  assumption  that  F'(Y  , t , 8 ) and  F" (Y  , t , S ) are 

P P P P P P 

distinct,  we  must  have  that  each  is  contained  in  a different  global 


1 
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Hamiltonian  hyperplane.  We  denote  these  global  Hamiltonian  hyperplanes 
by  H'(t^)  and  H" (t^)  respectively  and  express  them  as 


H'  (t  ):z'(t  ) = ■^.(t  )y.  + X > ^!(t  )y. 

P P X el  ^ P ^ X cB  ^ P " 

ip  1 P 


(4.14) 


H"(t  ):z"(t  ) = Lj  A.(t  )y.  + X)  ^"(t  )y.  (4  15) 

P P X el  ^ P ^ X eB  ^ P " 

ip  ip 


where  we  have  eliminated  the  destination  superscripting  since  we  are 

considexing  single  destination  networ)cs.  Now,  Y C H (t  ) and  from 

P P P 

Theorem  3.5,  part  (a),  we  )cnow  that  H (t  ) C H'  (t  ) and  H (t  ) C H"  (t  ) 

P P P P P P 

Therefore,  Y C H'(t  ) and  Y C H" (t  ).  T)ie  geometry  associated  with 
P P P P 

our  assumption  is  depicted  in  Figure  4.2. 

Consider  now  any  operating  point  e Y^.  Then  by  the  above  con- 
siderations we  Icnow  that  y C H'(t  ) and  y C H"(t  ) ; that  is,  y sat- 

p -p  p 

isfies  both  (4.14)  and  (4.15).  Substituting  into  both  of  these 
expressions  we  obtain: 


z'  (t  ) = z"(t  ) . 
P P 


(4.16) 


We  now  mal^e  the  further  assumptions  that  F'(Y  , t , 8 ) and 

P P P 

F" (Y  , t , B ) are  adjacent  faces  of  //.  We  may  do  this  without  any 
P P P ^ 

loss  of  generality  because  if  there  is  more  than  one  member  of 


•>'(Y^,  t^.  Bp)  we  may  always  pic)<  two  adjacent  members.  Now,  according 
to  the  definition  of  Lp-positive  face,  tliere  must  exist  some  point 
such  that 


Fiqure  4 . 2 


Geometry  in  ^-space  Associated  with 
Assumption  of  Non-uni>^ue  Costates 
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V e H' (t  ) n H" (t  ) 
P P 


(4.17) 


+ P 

where  ^ is  in  the  positive  orthant  of  R P;  that  is, 


y >0  V-x.cL  = 8. 

1 1 p p 


(4.18) 


By  subtracting  (4.15)  from  (4.14)  and  taking  into  account  (4.16),  we 


conclude  that  any  y e H'(t  ) O H"(t  ) satisfies 
^ p p 


E 

i.cB 
1 P 


(Af (t  ) - A"(t  ) )y.  = 0. 

1 P 1 p 1 


(4.19) 


Therefore,  ^ must  satisfy 


53  (>^.'(t  ) - X"(t  ))y'^  = 0. 

X.08  " P ^ P ^ 

1 P 


(4.20) 


Our  assumption  that  H' (t^)  and  H"(t^)  are  distinct  hyperplanes  implies 

that  X'(t  ) - A"(t  ) / 0 for  at  least  one  x.  e 6 . This  together  with 
1 P 3 P DP 

(4.18)  implies 

(A;  (t  ) - A"(t  ))y.  / 0 for  some  x.  c 6 . (4.21) 

3 P D P D 3 P 

But  (4.20)  and  (4.21)  imply  that  there  must  exist  some  x,  t 8 such 

k p 

that  A' (t  ) - AV (t  ) / 0 where  A'(t  ) - A"(t  ) is  of  opposite  sign 

kpKp  kpkp  ^ 

from  A)(t^)  - Xj(t^).  We  now  summarize  the  preceding  observations: 

Positive  Solution  Condition.  In  order  for  tlie  assumption  of 

non-unique  costates  to  be  true,  it  is  necessary  that 

A',  (t  ) - A'!(t  ) / 0 and  A'(t  ) - A"(t  ) 0 for  some  x C 8 

3 P 3 P k p k p 3 P 


r 
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and  X,  c B , Moreover,  it  is  necessary  that  Al(t  ) - XV (t  ) 
k p ] P 1 P 

is  of  opposite  sign  from  • 


Let  us  now  consider  the  region  F' (Y  , t , 8 ) O F" (Y  , t , 8 ) , As 

p p p P P P 

we  have  assumed  that  the  faces  F' (Y  , t , 8 ) and  F" (Y  , t , 8 ) are 

p p p P P P 

adjacent,  their  intersection  must  be  non-empty.  Also,  since  both 


P'(Y  , t , 6 ) and  F"(Y  , t , 8 ) are  n-1  dimensional  faces  of  the 

P P P P P P 

intersection  F' (Y  , t , 8 ) fj  F" (Y  , t , 8 ) is  an  n-2  dimensional  face 
P P P P P P 

of  that  is,  it  is  an  n-2  dimensional  convex  polyhedron. 


Recall  now  that  in  this  part  of  the  proof  we  are  assuming  that  Y^ 

contains  an  extreme  point  of  . Denote  such  an  extreme  point  by 

y e Y . By  Definition  3.9  of  an  L -positive  face,  we  have 
“^-e  p P 


e F' (Y  , t , B ) 
P P P 


and 
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y C F"(Y  , t , 8 ) . 

P P P 


(4.23) 


Therefore,  y e F' (Y  , t , 8 ) O F" (Y  , t , 8 ) . Since  y is  an  extreme 
■be  ppp  PPP  ^ 

point  of  ,y,  it  is  also  an  extreme  point  of  the  n-2  dimensional  convex 


polyhedral  face  of  F ' ( Y , t , 8 ) Pi  F"  (Y  , t , 8 ) . Tlierefore, 

PPP  PPP 

there  is  a minimum  of  n-2  independent  edges  of  F' (Y^,  t^,  8^)  ij 

F"(Y  , t , 8 ) which  emanate  from  y . These  are  also  edges  of  //.  Now, 
PPP  ^ 

we  have 


F'  {Y  , t , B ) C H’ (t  ) 
P P P P 


F"(Y  , t , B ) C H"(t  ) 
P P P P 


(4.24) 


in  accordance  with  the  geometrical  interpretation.  This  implies  that 

F'(Y  , t , B ) n F"(Y  , t , B ) C H'(t  ) n H"(t  ) (4.25) 

PPP  PPP  P P 


Hence,  in  order  for  our  assumption  of  non-unique  costates  to  be 
true  the  following  condition  must  also  hold: 


Edge  Condition.  If  ^ £ Y^  is  an  extreme  point  of  then 
there  exists  a minimum  of  n-2  linearly  independent  edges  of 
;i/  emanating  from  ^ which  are  also  contained  in 

H' (t  ) n H"  (t  ) . 

p p 

We  now  begin  the  task  of  providing  a contradiction  to  our  assump- 
tion by  demonstrating  that  the  positive  solution  condition  and  the 
edge  condition  cannot  hold  simultaneously.  In  brief,  we  shall  proceed 
by  enumerating  all  possible  types  of  edges  which  emanate  from  ^ and 
tliat  satisfy  the  requirements  of  the  positive  solution  condition  and  tlie 
edge  condition.  Our  conclusion  shall  be  that  the  maximum  number  cf 
such  edges  in  n-3,  which  is  one  short  of  the  number  required  by  tie 
edge  condition. 

Now,  the  two  basic  types  of  edges  of  ft  were  described  in  Section 


4.2  under  the  names  "Type  1"  and  "Type  2". 


In  brief,  Tyf>e  1 edges 
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involve  a change  in  exactly  one  coordinate  of  ^ while  Type  2 edges 

involve  changes  in  exactly  two  coordinates  of  We  now  wish  to  be 

more  specific  with  regard  to  the  type  of  state  variables  affected  by 

the  pivots / that  is,  whether  they  are  members  of  I or  B . Our  notation 

P P 

shall  be  that  a pivot  takes  us  from  the  current  operating  (extreme) 
point  ^ to  an  adjacent  extreme  point  We  denote  the  i^^  coordinate 

Q ♦-h 

of  ^ by  y^  and  the  i*"  coordinate  of  by  y!^. 


(i)  Type  1 edge  involving  ^ 

According  to  (4.8) -(4.9)  we  have 


, e 

y ■ = y ■ 

1 1 


y:  = y 


3 3 3 


V i j 

+ Ay.  where  |Ay.|  = C.,  > 0, 


(4.26) 


The  edge  condition  requires  e H'(tp)  O H"(t^).  Therefore,  the 
point  must  satisfy  the  equations  for  H'(t^)  and  H"(t^).  Substi- 
tuting into  (4.14)  and  (4.15)  and  using  (4.16)  we  obtain 


X . (t  ) (Ay  .)  + 2-f  (t  ) 

^ P 3 x^B  ^ P 


y]  x: (t  )y:  = 0 

^ 1 p 1 


(4.27) 


X . (t  ) (Ay .)  + 2^  X"(t  )y!  = 0. 

3 P 3 t P 1 


(4.28) 


x.eB 

1 P 


Taking  note  of  tlie  fact  that  y!  = 0 for  x,  C B (since  y.  = 0 for 

1 1 p 1 


X . e B ) we  obtain 
1 P 


X . (t  ) (Ay  .)  =0. 

3 P 3 


(4.29) 
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The  necessary  conditions  of  optimality  (3,32)  call  for  ) > 0 for 

J P 

X.  G I . But  since  (4.26)  specifies  |Ay,|  > 0,  equation  (4,29)  pre- 
3 P D 

sents  a contradiction.  Therefore,  this  type  of  edge  cannot  satisfy 
the  edge  condition  and  therefore  is  not  a candidate  for  our  n-2  required 
edges . 

(ii)  Type  1 edge  involving  x.  e 8 

I J £. 


In  this  case  we  have 


, e 

y = y ■ 
■'  1 1 


V-  i j 


y = Ay  , where  Ay. 

] 3 3 


(4.30) 


C.,  > 0 
3d 


By  (4.8) -(4. 9)  and  the  fact  that  y®  = 0 V-  e 8^.  Once  again,  the 
point  2.'  i^ust  satisfy  the  equations  for  both  H'(t^)  and  H"(t^).  Sub- 
stituting 2.'  into  (4.14)  and  (4.15)  and  simplifying  obtain 


X)(t  ) (Ay.)  = XV  (t  ) (Ay.)  = 0. 

3 P 3 3 P 3 


(4.31) 


But  jAy^l  > 0 and  (4.31)  imply 


X! (t  ) = X"(t  ) . 

3 P 3 P 


Therefore,  edges  of  this  type  are  potential  edges  of  ;*/  with  tlie 
stipulation  that 


X',  (t  ) - X"(t  ) = 0. 

3 P 3 P 


(4.32) 
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(iii)  Type  2 edge  involving  x . e 8 , x,  e 8 

3 P k p 

In  this  case  (4.1)  and  (4.11)  give 


y 1 = y? 


V i j,k 


y ! = Ay . 

^3 

= -Ay^  where  [Ay^]  = > 0. 


(4.33) 


As  before,  we  substitute  (4.33)  into  (4.14)  and  (4.15)  and  simplify  to 
obtain 


X((t  ) = XMt  ) 

3 p k p 

XV (t  ) = X"(t  ). 

3 P k p 


(4.34) 


(iv)  Type  2 edge  involving  x^  £ 8^,  x^^  e 
We  have  by  (4.10)  and  (4.11)  tliat 


, e 

y:  = y^ 


y '•  = Ay  . 
3 ^3 


V-  i ^ j,k 


''k  " ^k  " '^here  |Ay^| 


= C >0 

3k 


(4.35) 


I 


Substituting  (4.35)  into  (4.14)  and  (4.15)  and  simplifying  obtain 


X'.  (t  ) = X"(t  ) = 0. 

3 P 3 P 


(4.36) 


(v)  Type  2 edge  involving  x . e I , x,  C I 
3 E k 


From  (4.10)  and  (4.11)  we  have 


, e 

y = y- 

1 •*  1 

y ' = V®  + Ay  , 
D 3 3 


V-  i j,k 


(4.37) 


y'  = y®  - Ay  where  |Ay  | 


= C.,  > 0. 

3k 


Substituting  (4.37)  into  (4.14)  and  (4.15)  and  simplifying  obtain 


A.(t  ) = X,  (t  ). 

3 P k p 


(4.38) 


We  shall  refer  to  the  admissible  edges  represented  by  (ii)-(iv) 
above  as  "edges  of  type  (ii)-(iv)"  respectively. 

We  have  now  considered  all  possible  types  of  edges  emanating  from 
^ and  found  that  the  edge  condition  has  implied  certain  relationships 
among  the  costates  involved.  Now,  in  our  atten^jt  to  arrive  at  our 
contradiction  we  shall  not  attempt  to  explicitly  enumerate  all  edges 
emanating  from  ^ for  a specific  constraint  figure  ,>/.  Instead,  we 
shall  consider  all  possible  edges  of  types  (ii)-(iv)  above  which  may 
exist  and  show  that  there  are  no  more  than  n-3  independent  edges  which 
also  satisfy  the  positive  solution  condition.  In  order  to  facilitate 
the  discussion  concerning  an  arbitrary  set  of  edges,  we  introduce  the 
following  graphical  representation. 


Definition  4.10:  Consider  the  cotrponents  of  ^ (y^^,  ...,  y^)  to  be  the 

nodes  of  a graph  and  the  arrows  and  arcs  of  notation  4.1  to  be  edges  of 
a graph.  Then  an  edge  graph  is  the  collection  of  all  such  nodes  and 
edges  which  represent  a set  of  edges  of  ,'V  emanating  from 


Note  that  edges  of  Type  (ii)  (y^  where  e B^)  are  special  in 

that  they  only  involve  one  extremity  node.  We  now  divide  the  nodes  of 

an  edge  graph  into  two  mutually  exclusive  categories:  those  which 

correspond  to  state  variables  which  are  members  of  the  boundary  set  8^ 

and  those  which  correspond  to  state  variables  which  are  members  of  the 

interior  set  I . As  a shorthand  notation  we  shall  say  that  if  x.  e B 

then  y,  e 8 , where  B is  now  taken  as  a collection  of  nodes  of  the 
1 P P 

edges  graph.  Likewise,  if  x^  e Ip  we  say  that  y^  £ Ip.  The  use  of  the 

notations  8 and  I should  be  clear  from  the  context.  A typical  edge 
P P 

graph  categorized  in  this  fashion  is  presented  in  Figure  4.3,  where  we 
have  represented  components  of  ^ simply  as  y's. 


Figure  4.3  Typical  Edge  Graph 


Using  the  notation  we  have  established  we  now  reiterate  the  four 


possible  types  of  edges  which  may  exist  and  the  costate  conditions 


which  they 

imply : 

B 

P 

(ii) 

— ► X'.  (t  )-X',-(y  ) = 0 
IP  I P 

(4.39) 

6 

P 

(iii) 

) = X-(t  )-XVt  ) 

3P3P  kpkp 

(4.40) 

8 

P 

I 

P 

(iv) 

(4.41) 

I 

p 

(v) 

■ vv 

(4.42) 

Definition 

4.11: 

For  a given  edge  graph,  a 8^-subgraph 

(denoted  8*^, 
P 

where  the  integer 

valued  ot  ranges  over  the  number  of  8 - 

P 

■subgraphs) 

consists  of 

a set 

of  y's  in  6^  and  a set  of  edges  for  which  the  set 

of  y's  is  disconnected  from  all  other  members  of  8 . 

P 


Definition  4.12:  for  a given  edge  graph,  an  I -subgraph  (denoted  I^, 

P 

where  the  integer  8 ranges  over  the  number  of  -subgraphs)  consists 


disconnected  from  all  other  members  of  I . 

P 


Examples  of  8^-  and  I^-subgraphs  are  presented  in  Figure  4.3. 

cx  3 

Definition  4.13:  The  subgraphs  8 and  I are  said  to  be  connected  if 

^ ^ p p 

some  member  of  8 is  connected  to  some  member  of  I . 

P P 

For  example,  in  Figure  4.3,  8^  is  connected  to  I^. 

P P 

Ct 

Definition  4.14:  The  8^-subgraph  8^  is  said  to  be  independent  if  it  is 

not  connected  to  any  I^-subgraph. 

In  Figure  4.3,  the  B^-subgraph  8^  is  independent. 

We  now  assume  that  all  edges  of  any  subgraph  under  discussion 
satisfy  the  edge  condition.  Therefore,  the  relations  (4 . 39) - (4 . 42 ) 
are  in  force.  We  now  apply  these  relations  to  ma)ce  the  following  ob- 
servations concerning  members  of  8^-  and  subgraphs: 

• If  y.  e 8*^  aud  y c then 
3 P k p 


x:  (t  ) - X"<t  ) = AMt  ) - X"(t  ) 

DP  DP  k p k p 


(4.43) 


That  is,  for  any  members  of  the  same  B^-subgraph  the  values  of 
the  corresponding  costate  differences  are  the  same. 


•If  y.  e and  y e then 
DP  p 


X . (t  ) = X,  (t  ) . 
DP  k p 


(4.44) 


That  IS,  for  any  metobers  of  the  same  I^-subgraph  the  values  of 
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the  corresponding  costates  are  the  same. 


®If  y.  c 6^  and  y e and  and  8^  are  connected  then 
DP  p p p 

X.'  (t  ) - XV  (t  ) = 0.  (4.45) 

D P D P 

That  is,  each  member  of  a 8^-subgraph  which  is  connected  to  any 
Ip-subg^aph  has  its  corresponding  costate  difference  equal  to 
zero. 


• If  y.  e 

D P 


and  y^  has  the  type  (ii)  edge  y^,  then 
X:  (t  ) - X"  (t  ) = 0 V-  y.  e 

1 p 1 p •'i  p 


(4.46) 


That  is,  each  member  of  a 8^-subgrap)i  which  contains  a type  (ii) 
edge  among  any  of  its  members  has  its  corresponding  costate 
difference  equal  to  zero. 


Lemma  4.1:  In  order  that  the  positive  solution  condition  be  satisfied 

by  the  edges  represented  in  a particular  edge. graph,  it  is  necessary 
that  there  are  at  least  two  independent  8^-subgraphs  in  the  edge  graph, 
each  of  which  does  not  contain  a type  (ii)  edge. 


Proof  of  Lemma  4.1:  The  positive  solution  condition  requires  that 

we  have  an  x , C 6 and  an  x,  C 8 such  that  X'.  (t  ) - X'.'(t  ) 4 0 and 
DP  kp  Df'DP 

X^(t^)  - X^(t^)  4 0,  and  these  costatc  differences  mu.st  be  of  tlie 
opposite  sign.  Equation  (4.43)  tells  us  tliat  the  state  variables  whicli 
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satisfy  this  requirements  must  not  correspond  to  coordinates  of  ^ which 

are  members  of  the  same  B -subgraph.  Therefore,  x and  x must  corres- 

P J 

pond  to  members  of  separate  B^-subgraphs,  say  B^  and  b\  Now,  by  (4,45) 
neither  8-*  or  8 can  be  connected  to  any  I -subgraph,  and  hence  each  is 

p p ^ p ^ 

independent.  Furthermore,  (4.46)  implies  that  neither  8^  or  8^  contain 


a type  (ii)  edge. 


Lemma  4 . 1 


We  now  interpret  the  property  of  linear  dependence  of  edges  into 
our  graph  theoretic  approach. 

Lemma  4.2;  Any  subgraph  of  an  edge  graph  which  contains  a loop  must 
represent  a linearly  dependent  edge  of 

Proof  of  Lemma  4.2:  It  is  easily  seen  that  a two  node  subgraph  with  a 

loop  represents  two  linearly  dependent  edges  of  Now,  consider  the 

three  node  subgraph  with  a loop  in  Figure  4.4.  The  edges  are  ii’enti- 
fied  by  circled  integers.  Suppose  this  subgraph  represents  three  edges 


Figure  4.4  Three  Node  Subgraph  with  Loop 

( j ) 

of  .y  emanating  from  Let  us  now  denote  by  Ay^  the  change  in  the 

coordinate  y^  v;hich  occurs  when  moving  along  the  edge  ij)  from  ^ to 


an  adjacent  extreme  point  of  .*/ . From  (4.10)  and  (4.11)  we  tlien  liave 


the  following  relationships: 


Ay®=  -A® 

. (2)  (2) 

Ayr  = -Ay' 

A®  A® 

Ayj^  = -Ay3 


Let  us  now  c insider  the  three  vectors  in  j^-space  corresponding  to 
edges  (T)  , (2)  and  . When  we  perform  the  determinant  test  for 
linear  independence  of  these  vectors,  the  above  relationships  yield 


A® 

0 

0 

(2) 

A>, 

= 0. 

A® 

Ay^ 

0 

Therefore,  edge  (d)  is  linearly  dependent  upon 

edges  (T) 

and  @ 

Now,  consider 

a loop 

in  the 

a subgraph  of 

arbitrary 

size . 

Figure  4.5. 

^1 

V2 

^3 

y4 

^s-l 

/ 

. - - 

y 

• t 



/ 

' / 
/ 

/ 

N.  'V 

~ — 

- - 

Figure  4.5  Loop  in  Subgraph  of  Arbitrary  Size 
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R 


From  the  previous  argument  concerning  a three  node  loop  we  conclude 
that 


is  a linear  combination  of 


and  y„ 


'3' 


is  a linear  combination  of 


and  y y , 


yj^  y^  is  a linear  combination  of  y^^ 

Therefore,  the  edge  of  represented  by  y^ 
of  those  represented  by  y^  y^,  y^  ^3 ' ‘ 


y , and  y , y 
/s-1  s-1  £ 


y^  is  a linear  combination 


' ^s-1 


Lemma  4 . 2 


In  counting  linearly  independent  edges  of  an  edge  graph  we  shall 
be  considering  the  following  special  subgraphs. 


Definition  4.15;  A connected  subgraph  of  an  edge  graph  is  either  a 

B - 

P 


6 -subgraph,  an  I -subgraph,  or  the  union  of  a B -subgraph  and  an 
p P P 


Ip- subgraph  which  are  connected. 


Lemma  4 . 3 

(a)  Any  connected  subgraph  consisting  of  s nodes  has  s-1 
linearly  independent  edges,  not  including  any  edges 
of  type  (ii) . 

(b)  No  more  than  one  edge  of  type  (ii)  can  be  added  to  a 
connected  subgraph  representing  linearly  independent 
edges  of  and  have  the  new  set  of  edges  represented 
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remain  linearly  independent. 

Proof  of  Lemma  4,3; 

(a)  If  the  connected  subgraph  contains  a loop,  then  by  Lemma  4.2  it 
contains  at  least  one  linearly  dependent  edge.  Therefore,  discard  one 
edge  per  loop.  The  remaining  subgraph  is  loop  free  and  connected.  By 
Definition  4 0 it  is  then  a tree.  Also,  by  Proposition  4.4  it  contains 
s-1  edges.  It  is  not  difficult  to  see  that  all  of  the  edges  of  a tree 
are  linearly  independent. 

(b)  By  part  (a)  a connected  subgraph  with  s nodes  contains  s-1  linearly 
independent  edges.  But  we  may  have  not  more  tlian  s independent  edges 

in  a subspiice  of  dimension  s.  Therefore,  only  one  type  (ii)  edge  may 
be  added.  ■ Lemma  4. 3 


We  now  proceed  to  determine  the  maximum  number  of  independent 
edges  which  satisfy  Lemmas  4.1  and  4.3  (which  embody  the  positive  solu- 
tion condition  and  the  edge  condition) . The  breakdown  of  I and  8 is 

P P 

as  follows: 

According  to  Lemma  4.1,  there  must  be  at  least  two  independent 

8^-subgraphs  in  8^  each  of  which  does  not  contain  an  edge  of  type  (ii). 

Call  these  6 -subgraphs  8'  and  6".  Then  by  Lemma  4.3,  the  number  of 
P P P 

linearly  independent  edges  of  //  represented  by  these  B^-subgraplis 
are  cardinality  (8^^)-l  and  cardinality  (6^')-l  respectively. 

In  addition,  we  may  have  some  8^^-subgraphs , each  of  wliich  contains 
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n 


a type  (ii)  edge.  By  Lemma  4.3  each  of  these  represents  the  maximum 
possible  number  of  linearly  independent  edges  of  which  is  exactly 
the  number  of  nodes  in  the  subgraph.  Denote  the  collection  of  all 
such  subgraphs  by  8^'  . Then  the  number  of  linearly  independent  edges 
representedy  by  8^’  is  cardinality  (6^'  ) . 

Finally,  we  must  consider  all  of  the  I -subgraphs  and  the  remaining 

P 

B^-subgraphs . From  Theorem  4.3  we  conclude  that  these  subgraphs  will 
represent  the  maximum  possible  number  of  linearly  independent  edges 
of  fj  if  each  B^-subgraph  is  connected  to  an  I^-subgraph  by  an  edge  of 
type  (iv)  to  form  a connected  subgraph.  Denote  the  union  of  all  these 
subgraphs  by  'S . Then  the  number  of  linearly  independent  edges  repre- 
sented by  'fi  is  cardinality  ('S)-l. 

The  above  breakdown  is  depicted  in  Figure  4.6. 


I 

8 

p 

P 

8^  8^  8^  ...  8’  8"  8"' 

Figure  4.6  Edge  Graph  Ifliich  Satisfies  Positive 

Solution  Condition  and  Edge  Condition 


We  now  tally  up  the  total  maximum  number  of  linear  independent  edges  of 
•*/  represented  by  tiie  above: 


I; 
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I' 


I 


1 


mciximum  no.  of 
linearly  independent 

edges  = cardinality  (8'  + 8"  + 8'"  + '/^)  - 3 

P P P 

= cardinality  (8  U I ) - 3 
P P 


= n-3. 


(4.47) 


But  the  edge  condition  requires  that  we  produce  n-2  such  edges.  Hence, 


our  original  assumption  that  , t , IS  ) contains  more  than  one 

P P P 

element  must  be  false  if  Y contains  an  extreme  point  of  . 

P 


(b)  Yp  does  not  contain  an  extreme  point  of 


If  Y does  not  contain  an  extreme  point  of  ,*/,  then  it  must  lie  on 
P 

some  face  of  ,u  whose  dimension  is  greater  than  the  dimension  of  H (t  ). 

p p 

This  is  so  because  if  the  face  on  which  Y lies  is  of  dimension  less 


P 

than  or  equal  to  that  of  H (t  ) , then  Y must  be  that  face,  and  hence 

P P P 

contain  an  extreme  point  of  fy.  Let  tlie  g-dimensional  face  of  ,V/  on 


which  Y lies  be  denoted  F , where  a g < n-1.  See  Figure  4.7. 

P g P 

The  geometry  of  linear  programs  tells  us  that  any  globally  maxi- 


mizing Hamiltonian  hyperplane  H(t  ) which  contains  Y must  also  contain 


the  face  on  which  it  lies,  namely  F . Hence  H(t  ) must  contain  tlie 

g p 

q-1  dimensional  linear  variety  in  which  F lies,  call  it  H (t  ) . Now 

g g P 

H (t  ) must  contain  H (t  ),  so  that  it  has  tlie  form: 
g P P P 


✓ ^ X . (t  )y  . + 2^  ^ At  ) y 

x^I  ' ^ x^8-  ^ ^ 

i i.  1 


X.  ( 


1 


■ 

I 

I 

.j 


i 


] 

1 


j 

j 

'i 

i 

i 


H (t  ) : 

g p 


(4.48) 
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where  y?  is  the  component  of  any  point  in  and  6'  C The 

values  of  A , (t  ) for  all  x,  £ B*  are  now  considered  fixed.  The  global 
1 p 1 ^ 

Hamiltonian  hyperplane  must  have  the  form: 

H(t  ):  A,(t  )y.  + A.(t  )y.  + A.(t  )y. 

P x.el  ^ P " " P " x.eB”  " P " 


= S A.  (t  )y°  (4.49) 

x.el  ^ P ^ 

1 P 

where  B'  U 8"  = B . We  shall  once  again  proceed  by  contradiction.  We 

p p p 

assume  that  .jf(Y  , t , 8 ) contains  two  elements  and  we  associate  them 
P P P 

with  the  distinct  Hamiltonian  hyperplanes  H' (t  ) and  H"(t  ) whose 

P P 

equations  are  given  by 


H'(t  ):  X)  )y,-  + 5^  ^ )y.  + X]  AKt  )y. 

P . cT  ip  ^ ^ P 1 1 P 1 


x.el 
1 P 


= £ (t  )y° 

xTl  ^ P ^ 


(4.60) 


H"(t  ):  2^  A.(t  )y.  + 22  X.(t  )y.  + /2  A"(t  )y. 
P „ .7  1 P ^ —R.  1-  P 1 JTrz’.  1 P 1 


X . el 
1 P 


X , e.B ' ^ X , eB ' 

1 1 


= 2-j  A . (t  ) y^. 
x.el  ' P " 
i P 


(4.51) 


If  we  pick  any  extreme  point  of  F^,  say  we  are  assured  that  it  lies 

in  H'(t  ) n li"(t  ) with  11' (t  ) and  H"(t  ) as  given  in  (4.50)  and  (4.51). 
P P P P 
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Also,  ^ is  an  extreme  point  of  ;y.  Therefore,  we  may  proceed  as  in 
part  (a)  with  ^ replaced  by  replaced  by  U 8'  and  8^  replaced 


by  8".  The  statements  of  the  positive  solution  condition  and  the  edge 


condition  are  analogous  to  those  of  part  (a) . We  are  again  unable  to 
produce  more  than  n-3  edges  of  the  requires  type,  thus  contradicting 


the  assumption  that  , t , 6 ) contains  more  than  a single  element. 

P p p 


The  case  in  which  Y does  not  contain  an  extreme  point  of  is  com- 
P 


plete . 


The  proof  is  therefore  complete  for  the  case  in  which  ~ ^p' 


We  have  demonstrated  the  uniqueness  of  F(Y  , t , 8 ) . If  L C 8 , but 

p p p P P 


Lp  8p,  then 


a 


F(Y  , t , L ) = F(Y  , t , B ) D R ^ P 
p p p p p p 


where  the  basis  vectors  of  R ^ ^ are  the  elements  of  I 


p-1' 


There- 


fore, F(Y  , t , L ) is  the  unique  element  ot  , t , L ). 

p P P P P P 


Theorem  4. 1 


In  view  of  relation  (4.52)  and  the  fact  that  the  constructive 


algorithm  calls  for  every  state  variable  in  8^  to  leave  the  boundary 


bacltward  in  time,  we  shall  be  interested  in  F(Y  , t , 8 ) in  order  to 

P P P 


determine  all  the  leave-the-boundary  costate  values  at  the  potential 


boundary  junction  time  t^  e (-oo,  . For  the  purpose  of  the  current 


discussion  let  us  replace  t^  by  the  running  time  variable  t g 

Then  our  geometric  interpretation  of  the  necessary  conditions  has  the 


I 

J 
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global  Hamiltonian  hyperplane  H(T)  lying  on  the  unique  (by  Theorem  4.1) 

L -positive  face  F(Y  , T,  8 ) at  every  time  T C (-“,  t . Also,  we 
p P P P+1 

introduce  one  additional  piece  of  notation. 


Notation  4.2 


C = cardinality  (8  ) 
P P 


(4.52) 


Let  us  now  consider  the  case  in  which  Y contains  an  extreme  point 

P 

of  Call  this  point  Now,  in  order  for  all  state  variables 

in  8^  to  leave  the  boundary  backward  in  time  at  T,  we  must  be  able  to 
have  y^  ^ 0 for  all  C 8^.  Such  points  must  therefore  lie  on 
F(Y  , X,  6 ).  But  at  y we  have  y.  = 0 for  all  x.  £ 8 and  the  only 

p p ^ 1 1 p 

edges  which  involve  changes  in  y,  for  x.  c 8 are  those  of  types  (ii) , 

1 1 p 

(iii)  and  (iv)  detailed  in  the  proof  of  Theorem  4.1.  These  observations 
suggest  the  following  condition: 


Independent  Edge  Condition.  At  any  time  X , the  unique 

L -positive  face  F(Y  , x,  8 ) contains  C independent 
P P P P 

edges  emanating  from  of  either  type  (ii),,(iii)  or 
(iv)  detailed  in  the  proof  of  Theorem  4.1. 


I'or  reference,  we  now  list  the  edges  of  types  (ii)  , (iii)  and  (iv) 

and  detail  what  the  requirement  that  the  edges  lie  in  11  (x)  implies 

about  the  associated  costates.  First,  we  note  tliat  we  are  dealing 

with  a unique  global  Hamiltonian  which  contains  F(Y  , x,  8 ) at  time  x 

P P 
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and  express  this  as 


H(T):  A (T)y  + ^ ^ 

x.el  ^ x.eB  ^ ^ x.el  ^ ^ 

1 p 1 p 1 P 


(4.53) 


T e (-». 


. th 


where  is  the  i component  of  ^ evaluated  at  the  extreme  point  y^ . 
Utilizing  the  fact  that  every  edge  under  consideration  must  lie  in  the 
Hamiltonian  hyperplane  of  (4.53)  we  obtain  the  following  costate  rela- 
tionships corresponding  to  edges  of  types  (ii) , (iii)  and  (iv) : 

6 


(ii) 


X.(T)  = 0. 
3 


(4.54) 


(iii)  y.  y,. 

8 I 

P P 


(iv) 


X . (T)  = X (T) 
3 


X . (T)  = X (T) 
3 k 


(4.55) 


(4.56) 


We  now  shall  establish  what  the  edge  graph  will  look  like  at  a 
given  time  r in  light  of  the  independent  edge  condition.  We  make  the 
following  claim  and  then  justify  it: 


Edge  Graph  Characterization.  Every  S^-subgraph  contains 
an  edge  of  type  (ii)  or  is  connected  to  an  I^-subgraph 


J 
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with  an  edge  of  type  (iv) 


The  justification  for  the  above  assertion  is  as  follows:  8 con- 

P 

tains  members,  and  by  the  independent  edge  condition  it  must  contain 

independent  edges  of  type  (ii) , (iii)  or  (iv) . Now,  any  8^-subgraph 

containing  C members,  where  C < C , can  contain  no  more  than  C'-l 
P P - P P 

independent  ■ Iges  of  type  (iii)  by  Lemma  (4.1),  part  (a).  However,  by 

part  (b)  of  that  lemma  any  8^-subgraph  of  members  may  contain  no 

more  than  independent  edges.  Therefore,  every  8^-subgraph  must 

contain  a number  of  edges  equal  to  the  number  of  its  members.  This 

may  only  be  achieved  by  either  adding  an  edge  of  type  (ii)  or  connecting 

the  6 -subgraph  to  an  I -subgraph  by  an  edge  of  type  (iv) . 

We  denote  a 8 -subgraph  which  contains  a type  (ii)  edge  by  B^, 

P P 

where  the  integer  Y ranges  over  the  number  of  such  subgraphs.  The  edge 
graph  corresponding  to  this  situation  is  pictured  in  Figure  4.8. 


f 1 f 2 

8 

P P 


Figure  4.B  Edge  Graph  Which  Satisfies  Edge 
Graph  Characterization 
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For  convenience,  we  have  labelled  connected  8 - and  I -subgraphs  with 

P P 

the  same  superscript  in  Figure  4.8.  Before  continuing,  we  characterize 
edges  of  type  (v)  in  order  to  round  out  our  understanding  of  the  co- 
state implications  of  the  edge  graph. 


(v)  y.  y, 

^3  / 


yj(T)  = 


(4.57) 


We  now  return  to  consideration  of  some  y^  G 8^.  From  the  edge 
graph  characterization  above  we  see  that  y^  is  either  a member  of  some 

d ^ Y 

8 or  some  8 . We  now  examine  these  two  cases: 

P P 

U 

• If  y^  e 8p,  then  from  (4.54)  and  (4.55)  we  have 


A . (t)  = 0. 

1 


(4.58) 


•If  y^  e Bp,  then  from  (4. 55) - (4. 57)  we  have 


X.  (T)  = A . (T) 

1 D 


(4.59) 


for  any  y.  e where  is  the  I -subgraph  connected  to  8°^. 

D p P P P 


We  now  present  two  lemmas  which  together  characterize  the  dynamic 
costate  behavior  associated  with  single  destination  networl<  problems 
with  all  unity  weightings  in  the  cost  functional.  This  characteriza- 
tion will  constitute  the  basic  principle  in  the  proofs  of  the  subse- 
guent  results  of  this  chapter. 


I 


'■i 

1 
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Lennna  4.4:  Consider  single  destination  network  problems  with  all  unity 

weightings  in  the  cost  functional.  Suppose  all  of  the  costates  cor- 
responding to  state  variables  on  boundary  arcs  achieve  their  leave-the- 
boundary  values.  Then  every  costate  satisfies  exactly  one  of  the 
following  conditions  between  boundary  junctions: 


A. (T)  = A. (T)  = 0 (4.60) 

1 1 


or 

A . (T)  = -1  and  A . (T)  >0  (4.61) 

1 1 

VT  e (-00, 


Proof; 

We  begin  by  pointing  out  that  the  supposition  that  all  of  the 
costates  corresponding  to  state  variables  on  boundary  arcs  can  achieve 
their  leave-the-boundary  values  shall  be  shown  to  be  valid  in  the  dis- 
cussion following  Lemma  4.5.  Also,  we  make  the  assumption  of  Section 
3. 3. 2. 2 that  once  a state  variable  enters  an  interior  arc  it  remains  in 
the  interior  backward  in  time.  In  Theorem  4.4  we  shall  show  this 
assumption  to  be  valid  for  the  class  of  problems  currently  under  con- 
sideration. 

Consider  the  case  in  which  some  C I ^ . Then  by  the  assumption 
tliat  x^  remains  in  as  time  runs  to  minus  infinity  and  by  (3.32) 
we  have 


A. (T)  = 


p+1 


) . 


-1 


V I £.  (-«’,  t 
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We  must  also  have  X.(t  > 0,  so  A.(t)  satisfies  (4.61). 

1 p+1  1 

On  the  other  hand,  consider  the  case  in  which  some  e B^.  Since 
we  are  considering  the  leave- the-boundary  values  of  the  costates  cor- 
res5X3nding  to  state  variables  on  the  boundary,  our  geometric  interpre- 
tation has  H(T)  lying  on  the  unique  (-^-positive  face  F(Yp/  T,  8^)  at 
every  time  T e (-“>,  t ,).  See  Section  3. 3,2.2.  Recall  that  Y is 

p+1  p 

the  current  set  of  operating  points.  Uniqueness  is  insured  by  Theorem 

4.1.  Note  that  we  are  considering  the  global  Hamiltonian  hyperplane 

as  in  Theorem  4.1.  Now,  the  values  of  the  leave-the-boundary  costates 

at  T are  determined  by  the  face  F(Y  , x,  8 ) on  which  H(x)  lies.  But 

P P 

as  X runs  from  to  minus  infinity  H(x)  rotates  due  to  changing 

values  of  X.(x)  for  x.  e I . Therefore,  we  are  basically  concerned 
3 3 P 

with  investigating  if  B'(Y  , X,  8 ) changes  with  time,  and  determining 

P P 

what  is  implied  about  the  leave-the-boundary  costate  values.  To  this 

end,  we  now  divide  the  remainder  of  the  proof  into  two  sections  as  in 

the  proof  of  Theorem  4.1:  (a)  Y^  contains  an  extreme  point  of  and 

(b)  Y does  not  contain  an  extreme  point  of  A. 

P 


(a)  Y^  contains  an  extreme  point  of  A 


Denote  the  extreme  |x>int  by  , Then  by  the  independent  edge  con- 
dition stated  earlier,  K(Y  , r,  8 ) has  L independent  edges  emanating 

P P P 

from  . These  edges  are  represented  by  an  edge  graph  that  has  the 
format  of  Figure  4.8.  I’he  leave-the-boundary  values  of  the  costates 
at  X are  determined  by  the  edge  graph  at  X through  (4.58)  and  (4.59). 
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We  therefore  wish  to  examine  the  time  varying  behavior  of  the  edge 
graph. 

By  the  edge  graph  characterization  we  know  that  the  edge  graph 
must  maintain  the  format  of  Figure  4.7  as  a function  of  time.  We  must 
consider  both  the  situation  in  which  the  edge  graph  does  not  change  with 
time  and  that  in  which  it  does  change  with  time.  The  latter  situation 
must  correspond  to  the  disappearance  of  a set  of  edges  and  the  appear- 
ance of  an  equal  number  of  edges  elsewhere.  We  shall  refer  to  the  time 
at  which  such  an  edge  change  occurs  as  an  edge  switch  time,  denoted  by 
T^.  Also,  the  times  just  before  and  after  the  switch  (in  backward 
time)  are  denoted  and  respectively. 

We  now  return  to  our  discussion  of  some  state  x,  e B and  examine 

I P 

the  behavior  of  A^(t)  as  determined  by  the  edge  graph  behavior  as  a 
function  of  T.  The  situations  to  be  considered  are: 


1)  No  switch  in  edge  occurs. 

2)  A switch  in  edge  occurs  at  x ^nd  y.  is  in  some  8 before 

SI  p 


the  switch  (that  is,  at  x = Tj.)  • 


3)  A switch  in  edge  occurs  at  x and  y.  is  in  some  B before 

s i p 

the  switch  (that  is,  at  t = T^) , 

s 


We  now  consider  the  above  situations  one  at  a time: 


1)  No  switch  in  edge  occurs. 

>1  lx 

If  y.  r B for  some  subgraph  8 , tlien  X.  (t)  =0  for  all 

i p P t 

X t (-00,  tp^X^  (4.50).  In  this  case  condition  (4.60)  of  the  lemma 
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“l 

statement  is  satisfied.  On  the  other  hand,  suppose  e for  some 


“1 


subgraph  8 . Then  X.(T)  = X.(T)  for  any  y.  e I , where  I is  the 

P i 2 2 P P 

Ct 

subgraph  connected  to  8^.  Therefore,  condition  (4.61)  of  the  lemma 
statement  is  satisfied. 


2)  A switch  in  edge  occurs 


s '1 

The  possibilities  for  this  case  are: 


at  T and  y.  £ o at  T = T . 


• T = T ;y . e R 
S I p 

T = T :y,  e 8 
s ■'i  p 

Here  we  are  considering  the  situation  in  which  the  edge  switch 
causes  the  composition  of  8^  to  change  but  that  y^  remains  a member. 
Now,  from  (4.58)  we  have 


X . (T)  =0 
1 

X^(T)  = 0 


" ‘^s'  Vl’ 

V-  T C (-“,  T“) 


(4.62) 


so  that  X.  remains  equal  to  zero  on  (-“,  t ,). 

1 p+1 


• T = T'^iy . c 8 ^ 
S I P 


T = T”:y.  e 6 ‘ 
s •'  1 p 


Y / Y 
1 ^ '2 


Once  again,  we  apply  (4.58)  to  obtain 


X.(T)=0  V-xe(T,t  ,) 

1 s p+1 

X^(t)  =0  V T C (-",  t”) 


(4.63) 


so  that  X^  remains  equal  to  zero  on  (-“',  • 
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+ ' n 

• X = X^:y . e 8 ^ 

s ■'i  p 


X = x":y.  e r 

S I p 


Equations  (4.58)  tells  us  that 


X.  (X)  = 0 V-  X c (X^,  t 
1 s p+1 


(4.64) 


and  (4.59)  says 


X.(x)  = X.(x)  >0  V X e (-0°,  x„) 


(4.65) 


where  y^  C I^.  But  (4.64)  and  (4.65)  imply  that  X^  experiences  a jump 
of  negative  sign  forward  in  time  across  X^.  But  this  contradicts  the 
fact  that  the  costates  may  only  experience  positive  jumps  forward  in 
time  by  equation  (3.36).  Therefore,  this  )<ind  of  edge  switch  cannot 


Summarizing  the  above  situations,  we  see  that  if  y^  is  in  some 

fYi 

Bp  at  any  time  x on  (-m,  tlien  X^  remains  zero  on  this  interval. 

This  behavior  suits  equation  (4.60)  of  the  lemma  statement. 

“l  + 

2)  A switch  in  edge  occurs  at  i and  y.  e B at  t = x • 

6 X p S 

We  begin  with  the  following  observation  which  results  directly  from 
the  fact  that  the  Hamiltonian  hyperplane  H(x)  rotates  continuous!'/  in 


Edge  Switch  Condition:  Let  ig  bo  the  time  at  wtiich  the 

edge  graph  switcli  occurs.  Suppose  tlie  edge  e^  is  replaced 
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This  corresponds  to  the  situation  in  which  a member  is  added  or 

removed  from  8 , From  (4 . 55) - (4 . 57)  we  have 

P 

X (T)  = A.  (T)  V-  T e (t"^,  t ) 

1 j s P+1 

X . (T)  = X . (T)  V-  T e (-«,  T~) 


1 

For  any  y . c I . Therefore 
D P 


X (T)  = X . (T)  Vie  t ^, ) . 

1 3 P+1 


(4.66) 


With  the  assumption  that  states  on  interior  arcs  remain  on  interior 
arcs  as  T runs  to  minus  infinity,  we  have  from  (3.32)  that 


and 


X.  (t  ) >0 

3 P+1 

X . (T)  = -1  V-  T e (-“,  t ) 
3 P+1 


(4.67) 


Equations  (4.66)  and  (4.67)  imply 


and 


X.  (T)  >0 
1 

X.  (T)  = -1  ¥ T e (^,  t ^,  ) 

1 P+1 


(4.68) 


T = T^:y.  e 8 ^ 
s ■'  1 p 

“2 

T = T :y,  e 6 
S I p 


“1  ^ “2 


Then  from  (4 . 55) - (4 . 57)  we  have 


X.  (t)  = X . (t)  t C (t  , t ) 

13]^  s p+1 


X.  (t)  = X . (i) 
1 I5 


(-,  t^) 


(4.69) 


V-  T e 
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1 2 

for  any  y . e I and  y . E I . But  the  edge  switch  condition  says 

P ^2  p 

that  at  the  time  of  switch  T , y.  e B and  y,  E B . Consequently, 


s - 1 


by  (4.69)  we  have 


X.  (T  ) = X . (T  ) = X . (T  ) 
IS  s 32  ® 


(4.70) 


Once  again  invoking  the  assumption  that  states  that  interior  arcs 
remain  on  interior  arcs  for  all  T E (-“,  t ,.) , we  have 

p+1 


X.  (T)  = X.  (T)  = -1  VT  E (-“,  t .) 


p+1' 


(4.71) 


and 


X.  (t  ) > 0. 

3i  P+1 


Equations  (4 . 69) - (4 . 71)  imply 


X^(T)  > 0 


X (T)  = -1  V-  T E (-“,  t . 

1 p+1 


+ “l 

• T = T_:y . E 

SI  p 


(4.72) 


T = T_:y-  E 8' 

SI  p 


The  edge  switch  condition  says  that  botli  of  the  above  edges  are  in 

effect  at  switch  time  x • Under  this  circumstance,  equation  (4.58) 

s 

tells  us  that 

X. (x  ) = 0 (4,73) 

X s 


and  (4.59)  says 
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X.  (T  ) = X . (T  ) >0 

IS  US 


(4.74) 


1 

for  some  y . £ I 
3 P 


Since  (4.73)  and  (4.74)  are  in  contradiction,  this  )cind  of  edge 
switch  cannot  occur. 


Summarizing  the  above  situations,  we  see  that  if  is  in  some 


at  any  time  T £ (-“,  , then  X^(T)  = -1  and  X^(T)  > 0 for  all 


T e (-“,  • This  behavior  suits  equation  (4.61)  of  tlie  lemma  state- 


ment. Hence,  the  lemma  is  proved  if  contains  an  extreme  point  of  M 


(b)  Yp  does  not  contain  an  extreme  point  of  fy 


We  talie  tiie  same  approach  here  as  in  part  (b)  of  the  proof  of 


Theorem  4.1.  The  solution  set  Y lies  on  some  face  of  fy,  call  it  F , 

P g 


of  dimension  g,  where  ^ 9 ^ n-1.  Refer  to  Figure  4.7.  Following 


the  reasoning  of  that  section,  we  now  have  the  global  Hamiltonian 


nil):  + S 

x.el  x.cB'  ^ x.cS"  ^ ^ 


1 P 


1 P 


i P 


^ (T)y . 
x.cl  ^ ^ 

1 P 


(4.75) 


where  B'  U B"  = B . If  we  now  pic)c  any  extreme  point  of  F , say  y'  , 
P P P g'  ^ 

may  proceed  as  in  part  (a)  of  this  proof  with  replaced  by  y^,  B^ 


we 


replaced  by  B|^  and  the  edges  corresponding  to  B^  fixed  (since  these  are 


the  edges  of  the  invariant  face  F^  upon  which  Y^  lies) . We  are  again 


J 
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able  to  show  that  in  the  face  of  all  possible  switches  in  edges,  the 
costates  behave  either  as  in  (4.60)  or  (4.61).  ■Lemma  4.4 

Now,  statements  (4.60)  and  (4.61)  of  Lemma  4.4  imply  that  the 

costate  trajectories  on  (-“,  t , ) appear  as  in  Figure  4.10. 

P+-*- 

Lemma  4.5;  Consider  single  destination  network  problems  with  all 
unity  weightings  in  the  cost  functional.  Suppose  all  of  the  costates 
corresponding  to  state  variables  on  boundary  arcs  achieve  their  leave- 
the-boundary  values.  Then  the  costates  do  not  experience  jumps  across 
boundary  junction  times. 


Proof: 


As  in  the  case  of  Lemma  4.4  we  point  out  that  the  supposition  that 
all  of  the  costates  corresponding  to  state  variables  on  boundary  arcs 
can  achieve  their  leave-the-boundary  values  is  shown  to  be  valid  in  the 
discussion  following  this  proof. 

Consider  now  the  boundary  junction  time  t^  £ (-“,  which 

the  state  variables  in  L leave  the  boundary  backward  in  time.  Then 

P 

for  T C [t  , t ,,)  the  set  of  state  variables  on  the  boundary  is  B , 
p P+1  P 

the  set  of  states  in  the  interior  is  and  the  ^-space  solution  set 
is  Y . Likewise,  for  T e (-“,  t ) those  sets  are  B , , I , and  Y 

p p p-1  p-1  p-1 

respectively. 

Now,  consider  some  x.  e I . Then  by  (3.22)  X.  does  not  experience 
^ E.  1 

a jump  at  t^. 


J 
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Consider  now  some  e B^.  If  e L^,  then  by  hypothesis 
is  at  its  leave-the-boundary  value.  Therefore,  no  jump  in  is  exper- 


ienced when  x^  leaves  the  boundary.  On  the  other  hand,  suppose 

X.  e 8 /L  , that  is,  x.  c 8 . in  this  case  X.  is  determined  by  the 

1 p p 1 p-1  1 

face  F (Y  ,,  t , 8 , ) on  which  H(t  ) lies.  We  now  claim  that 

p-1  p p-1  p 


F(y  - , t , 8 ) = F(Y  , t , 8 ) , 

p-1  p p-1  P P P 


(4.76) 


To  verify  (4.76)  we  first  note  that  by  the  continuity  of  the  Hamilton- 


ian, the  solution  set  Y , lies  on  F(Y  , t , 8 ).  Also,  by  definition 

p-1  P P P 

F(Y  , t , 8 ) is  the  8 -positive  face  of  iO  with  respect  to  Y and 
P P P P P 

H(t  ).  Therefore,  since  8 CB,F(Y,t,B)  must  also  be  the 
P P-1  P P P P 

8 .-positive  face  of  ;'/  with  respect  to  Y , and  H(t  ).  The  preceding 
p-1  p-1  P 

statement  is  equivalent  to  (4.76).  As  a consequence  of  (4.76)  we  have 


that  X . for  x.  e 8 , does  not  change  across  t . This  completes  the 

X 1 p-1  ^ p 

proof  that  no  costate  jumps  across  t . ■ Lemma  4.5 


We  conclude  from  Lemma  4.5  that  across  the  boundary  junction  t^ 
all  the  costates  continue  to  propagate  backward  in  time  as  in  Figure  4.10 
except  for  the  following  cases: 

(i)  X.(t)  =0  for  T e (t  , t ,)  and  x.  is  leaving  the 

X p p+1  X 

boundary  at  t . In  this  case  X.(t)  = -1  for  T C (-“,  t ). 

P X p 


(ii)  X.(T)  = 0 for  T E (t  , t ,)  and  x.  remains  on  the 

X p p+1  X 

boundary  but  the  leave-the-boundary  value  of  X^(t), 
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T e (-»,  t ),  dictates  that  X.(T)  = X.(T)  for 
P 1 3 

some  X^  satisfying  case  (i)  above.  In  this  case 

X.(t)  = X.(T)  = -1  for  T e (-“,  t ). 

1 3 P 

The  picture  of  the  costate  behavior  across  the  boundary  junction  time 
is  presented  in  Figure  4.11,  Note  that  the  set  of  satisfying  (i) 
and  (ii)  abc'e  are  not  dependent  upon  the  boundary  junction  time  t^. 

We  now  combine  the  results  of  Lemmas  4.4  and  4.5  into  a general 
description  of  the  costate  trajectories.  Consider  any  backward  optimal 
trajectory  corresponding  to  some  sequence  of  state  variables  leaving 
the  boundary  backward  in  time  at  a series  of  boundary  junction  times. 
Then  if  all  boundary  costates  achieve  their  leave-the-boundary  value, 
then  each  costate  X^  satisfies  the  following  trajectory: 


X (T)  = X,  (T)  = 0 

V-  T E (t.  , t,) 

1 1 

1 f 

X^(T)  = -1 

V-  T e (-“,  t^) 

(4.77) 


where  t^  is  one  of  the  boundary  junction  times  between  (and  possibly 
including)  t^  and  the  time  at  which  leaves  the  boundary.  A typical 
set  of  costate  trajectories  for  a backward  optimal  trajectory  is 
depicted  in  Figure  4.12. 

In  Lemmas  4.4  ar.d  4.5  we  make  the  supposition  that  all  Ixjundary 
costates  can  acliieve  t)ieir  leave-the-boundary  values.  We  now  verify 
that  supposition.  In  (4.77)  we  describe  the  trajectory  which  must  be 
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satisfied  by  any  costate  in  order  that  all  boundary  costates  are  main- 
tained at  their  leave-the-boundary  values.  The  required  trajectories 
bear  the  following  relationships  between  boundary  and  interior  arcs 
(see  (4.77)): 

( 

V X. (T)  = 0 < or  (4.78) 

( A^(T)  = 0 

V X. (T)  > 0 X. (T)  = -1  (4.79) 

1 1 

Since  = 1 for  all  x^,  then  (4.78)  and  (4.79)  satisfy  the  necessary 
conditions  (3. 32) - (3. 34) . Therefore,  the  leave -the -bo undairy  values  of 
tlie  costates  can  be  achieved  by  costate  trajectories  which  satisfy  the 
necessary  conditions. 

Lemma  4.6;  The  set  of  optimal  controls  does  not  switch  between  boundary 
junction  times;  that  is,  there  are  no  brea)c  points  between  boundary 
junctions. 

Proof : In  accordance  with  the  necessary  conditions  (including  those 

satisfying  the  constrained  minimization) , all  optimal  controls  must  be 
a solution  to  the  global  minimization 

u*(T)  = ARG  MIN  a"^(t)x(t)  (4.80) 

ucU 

pointwise  in  t for  all  x e [t^,  t^) . Consider  the  typical  control 

variable  u for  a single  destination  network.  Since  u,.  enters  into 
1 k IK 


r 
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either  oj.e  or  two  state  dynamic  equations  in  the  fashion  of  equations 

T 

(4.6)  and  (4.7)  respectively,  then  the  term  of  ^ (t)x(t)  which  involves 

u. , is 
ik 

(X.  (T)  - A.(t))u..  (T)  (4.81) 

k 1 ik 

where  we  take  A,  = 0 if  node  k is  the  destination.  The  constraint 
k 

associated  with  u is  simply  0 < u.,  . The  optimal  control  is 

XK  XiC  XK 

then 


*^ik 

if 

A^(x) 

- A^(x) 

< 0 

U*.  (X)  = I 

0 

if 

A^(x) 

- A^(x) 

> 0 

(4.82) 

ik  ^ 

if 

A^(x) 

- A^(x) 

= 0 

V (i,k)  £ L. 

The  optimal  control  therefore  depends  only  on  SGN(Aj^(T)  - A^(t)).  But 

from  Lemma  4.4  we  see  that  SGN(A,  (x)  - A.(x))  remains  constant  on 

k 1 

(-“>,  if  state  variables  leave  the  boundary  on  that  interval; 

that  is,  SGN(Aj^(T)  - A^(x))  is  constant  between  boundary  junction  times. 
Consequently,  the  set  of  optimal  controls  does  not  switch  between 
boundary  junctions.  ■Lemma  4.6 

From  Lemma  4.6  we  know  that  for  the  class  of  problems  under  con- 
sideration all  feedback  control  regions  are  of  the  non-break  variety. 

Theorem  4 . 2 

For  single  destination  network  problems  with  all  unity  weiglitinqs 


in  the  cost  functional,  tliere  is  one  suliregion  per  region  witJi  respect 
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to  any  set  of  state  variables  leaving  the  boundary. 


Proof; 

Consider  the  feedback  control  region  constructed  from  the 

optimal  solution  on  (-“,  t Also,  let  L be  the  set  of  state 

P+1  P 

variables  which  we  choose  to  allow  to  leave  from  backward  in  time, 

P 

where  c:  According  to  Lemma  4,6,  the  optimal  trajectory  asso- 

ciated with  the  state  variables  in  leaving  backward  in  time, does 
not  experience  a break  as  time  runs  to  minus  infinity.  Denote  by  ^ 
the  single  set  of  optimal  controls  corresponding  to  the  trajectory. 

Then  according  to  Definition  3.8  we  must  show  that  the  state  variables 
in  leave  with  the  same  set  of  optimal  controls  from  every  point 
in  ,>f^.  This  is  equivalent  to  showing  that  the  set  of  optimal  controls 
associated  with  the  state  varic±>les  in  leaving  the  boundary  is 
the  same  for  every  boundary  junction  time  t^  e (-“,  • Note  that 

we  need  not  be  concerned  with  the  common  breakwall  stipulation  of 
Definition  3.8  since  there  are  no  breakwalls. 

We  begin  by  noting  that 


r'l 


(4.83) 


where 


is  the  set  of  optimal  controls  with  which  the  state 


variables  in  L leave  backward  in  time  and  that 
P P 

keep  the  state  variables  in  8 , 


on  the  boundary, 
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I 

'u 

f 


f 

i 


1 


Q. 


G 


is  the  globally  optimal  set 
with  the  state  variables  in 
in  time, 


of 

L 

P 


controls  associated 

leaving  backward 
P 


and 


P-1 


is  the  set  of  all  controls  which  keep  the  state 
variables  in  8^  ^ on  the  boundary. 


Now,  is  time  invariant.  Therefore,  if  we  can  show  that  is 

the  same  for  any  t e (-°°,  t ^.),  then  the  desired  result  is  obtained. 

P p+i 

Q 

To  this  end,  recall  that  SI  is  given  componentwise  by  (4.82)  where 
T e (-“,  t ).  From  Lemmas  4.4  and  4.5  we  conclude  that  SGN(X  (t)  - 

r K 

A.  (T))  is  the  same  for  all  T e (-",  t ) and  all  t e (-”,  t ,).  This 
1 P P p-1 

observation  coupled  with  (4.82)  give  us  the  desired  result. 


■ Theorem  4 . 2 


Theorem  4 . 3 

For  problems  involving  single  destination  networks  with  all  unity 
weightings  in  the  cost  functional,  any  solution  to  the  constrained 
optimization  problem  (of  Operation  1,  part  (a))  is  also  a globally 
minimizing  solution. 

Proof:  Let  us  assume  that  Y is  a set  of  o -dimensional  ooeratino 

P p 

points  which  maximizes  the  restricted  Hamiltonian  hyjxirplane  H (T) 

with  respect  to  the  restricted  constraint  figure  //  at  time  t.  In 

P 

Section  3. 3. 2.2  we  identified  the  following  demonstration  of  global 
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optimality  of  Yp  at  T;  Find  values  of  A^(t)  for  e such  that 

H(T)  is  maximized  over  fj  at  Y^,  that  is,  values  which  rotate  the  global 

Hamiltonian  hyperplane  H(T)  about  H (T)  until  H(t)  becomes  tangent  to 

P 

iy  at  Y^.  In  the  discussion  following  Lemma  4.5  it  was  established 

that  values  of  the  boundary  costates  can  always  be  found  which  bring 

H(t)  tangent  to  F(Y  , T,  8 ).  Consequently,  H(T)  may  be  brought  tan- 
P P 

gent  to  since  F(Y  , T,  8 ) is  a face  of  ;*/.  ■ Theorem  4.3 

P P 

The  following  theorem  is  the  final  result  of  this  section.  It  is 
equivalent  to  saying  that  there  always  exists  a solution  which  does  not 
call  for  the  return  of  state  variables  to  the  boundary  backward  in  time. 
Hence,  the  assumption  made  in  the  beginning  of  the  proof  of  Lemma  4.4 
is  valid. 


Theorem  4.4 

For  problems  involving  single  destination  networks  with  all  unity 
weightings  in  the  cost  functional,  there  always  exists  an  optimal  con- 
trol which  has  x.  <0  for  all  x,. 


Proof : Suppose  that  x^  leaves  the  boundary  backward  in  time  at  as 

a member  of  some  Then  x^  leaves  the  boundary  with  some  optimal 

slope 


X.  =b.u  +a.  <0 
1 —1“?  1 


(4.84) 


where  b^  is  the  i^^  row  of  B_  and  is  an  optimal  control  which  applies 
on  tp^j^)  if  no  other  state  variables  leave  the  boundary  backward 
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in  time.  We  shall  now  show  that  x*  given  by  (4.84)  remains  optimal  on 
t , ) even  if  other  state  variables  leave  the  boundary  bac)cward  in 

p+1 

time  on  that  interval.  This  property  holds  no  matter  what  sequence  of 
states  leave  the  boundary.  Suppose  that  the  state  variables  in 
leave  the  boundary  at  some  boundary  junction  time  t^  < ''p+1' 

Figure  4.13.  Let  us  now  examine  the  optimal  values  of  the  controls 
which  enter  into  xj.  Consider  the  typical  control  variables  u^^,  and 
Ujj^,  where  u^j^  represents  those  control  variables  on  links  outgoing 
from  node  i and  Uj^^  represents  control  variables  on  links  incoming  to 
node  i.  Then  the  terms  to  be  minimized  corresponding  to  these  control 


variables  are 


a.  (T)  - A.  (T))u.. 
k 1 rk 


where  T e (t  , t , ) 
P P+1 


(4.85) 


Now  since  x.(t)  > 0 for  T e (t  , t .,)  then  A,(x)  >0  for  T e (t  »t  ) 
1 p p+1  1 p p+1 

We  begin  by  considering  u^j^  and  the  following  two  possibilities: 


(i)  ” A^(t)  / 0 for  T c [t^. 

Tiien  tlie  optimal  value  of  u,,  on  [t  , t , ) is 

^ ik  p p+1 


P-1  Xi  I' 

y^Z/ZZr/-^7///yi/Z77/77Z/Z/Z77/^// 


p 


Figure  4.13  State  Tra]ectories  for  Proof  of 


, 
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But  from  (4.77)  we  know  that  SGN(X  (T)  - X,  (t) ) is  constant  for 

k 1 


T e (-00,  irregardless  of  which  state  variables  leave  the  boundary 


on  that  interval.  Therefore,  the  control  in  (4.85)  is  optimal  on 


T e (-°°,  irregardless  of  which  state  variables  leave  the  boundary 


on  that  inti  val. 


(ii)  X (T)  - X.(T)  = 0 for  T c [t  , t ^,) 

k 1 p p+1 


Then  the  optimal  value  of  u.,  on  [t  , t , ) is  any  value 

ik  p p+l 


0 ^ ^ which  together  with  all  the  other  controls  keeps  the 


state  variables  in  6^  on  the  boundary.  But  by  (4.77)  and  the  fact  that 


X . (t  ) > 0 for  i e (t  , t 1 ) we  conclude  that  X (x  ) - X.(x)  = 0 for 

1 p p+1  k 1 


dll  r £.  (-^®,  irregardless  of  which  states  leave  the  boundary  on 


ttiat  interval.  Therefore,  if  we  consider  the  interval  [t  ,,  t ),  then 

p-1  p 


t)ie  optimal  value  of  u.,  on  [t  , , t ) is  any  value  0 < u.,  < C.,  which 

ik  p-1  p ■'  — ik  — ik 


together  with  all  the  other  costates  keeps  the  state  variables  in  8 


p-1 


on  the  boundary.  But  since  8 , CT  g any  optimal  control  on  (t  ,t  ,) 

p-1  P P P+1 


which  keeps  the  state  variables  in  8^  on  the  boundary  must  also  be  an 


optimal  control  on  (t  ,,  t ) which  keep  the  state  variables  in  8 .on 

j)+l  p ^ p-1 

the  boundary.  Therefore,  by  induction  on  all  possible  time  intervals 
backward  in  time  corresponding  to  state  variables  leaving  the  boundary 


we  may  conclude  that  the  optimal  value  of  u^^^  on  [t^,  lp+^^  optimal 


for  all  r c (-”,  t , ) . 

P+1 


1 

'I 
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Arguments  (i)  and  (ii)  above  also  pertain  to  Consequently, 

for  any  control  variaible  which  enters  into  the  equation  for  we  have 
that  any  optimal  value  for  T e [t^,  is  also  an  optimal  value  for 


all  T e ( 

X e (-“,  t 


t^j^)  . We  conclude  that  x^  is  optimal  for  all 


P+1 


) . 


Theorem  4.4 


4.4  Computational  Algorithm 

In  light  of  the  results  of  Section  4.3,  we  are  able  to  construct  a 
computer  algorithm  which  realizes  the  backward  constructive  algorithm 
of  Chapter  3 for  the  special  case  under  consideration.  We  begin  by 
making  the  following  observations: 


• By  Theorem  4.1,  the  leave-the-boundary  value  of  the  costates 

are  unique  and  given  by  the  coefficients  of  the  global  Hamiltonian 

hyperplane  which  lies  on  the  B^-positive  face.  Therefore,  all  we  need 

do  to  determine  these  costates  at  some  boundary  junction  time  t^  is 

to  rotate  H(t  ) about  H (t  ) until  it  touches  F(Y  , t , 8 ).  We  shall 
P P P P P P 

demonstrate  a technique  for  this  shortly. 


•Suppose  we  have  a set  of  state  variables  off  the  boundary  as 
we  are  working  our  way  backward  in  time  on  an  optimal  trajectory.  Then 
the  rule  for  the  execution  of  the  basic  step  (see  Section  3. 3. 2. 2)  calls 


for  allowing  the  members  of  L to  range  over  all  possible  subsets  of  6 

P P 

at  the  boundary  junction  time  t^.  Now, from  Theorem  4 . 2 we  realize  that 

t may  be  any  time  on  (-«>,  t , ) so  that  we  may  arbitrarily  take 
P P+I 


•I 
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t = t - 1.  Furthermore,  Theorem  4.3  tells  us  that  each  of  the  sets 
P P+1 

of  state  variables  leaving  the  boundary  corresponds  to  a globally  opti- 
mal solution,  so  that  the  test  for  global  optimality  (Section  3. 3. 3. 2, 
Operation  1,  part  (b) ) is  not  required. 

• Consider  a particular  set  of  state  variables  leaving  the 
boundary  at  some  boundary  junction  time  t^.  Then  the  initial  set  of 
controls  corresponding  to  this  situation  persists  to  time  equals  minus 
infinity  without  a break  according  to  Lemma  4.6.  Therefore,  we  need 
only  perform  the  constrained  optimization  problem  once  on  (~°°,  t^) , so 
that  we  may  arbitrarily  take  the  time  at  which  it  is  performed  as  t^ 
Moreover,  every  feedback  control  region  constructed  is  of  the  non-break 
variety  of  Section  3. 3. 3.3. 

•Theorem  4.4  tells  us  that  no  state  variable  is  ever  required  by 

optimality  to  increase  forward  in  time.  Therefore,  if  we  consider  only 

• j -i 

those  trajectories  for  which  x^(t)  ^ 0 for  all  and  all  t,  then  we 
shall  never  have  state  variables  returning  to  the  boundary  backward  in 
time. 

Now,  the  second  observation  above  says  that  in  order  to  be  certain 
that  we  will  fill  up  the  state  space  with  optimal  controls,  we  must 
allow  all  possible  combinations  of  state  variables  to  leave  the  bound- 
ary in  all  possible  orders.  Furthermore,  we  are  guaranteed  that  all 


I 


of  these  sequences  of  state  variables  leaving  the  boujidary  will  corres- 
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pond  to  optimal  trajectories.  As  pointed  out  in  Section  3. 3. 2. 2,  by 
allowing  all  possible  combinations  of  state  variables  to  leave  the 
boundary  we  are  being  conservative  and  will  probably  be  "writing  over" 
previously  constructed  feedback  control  regions  on  occasion.  At  any 
rate,  by  the  final  observation  above  we  may  list  a priori  all  of  the 
appropriate  sequences  where  the  departure  of  the  state  variables  from 
the  boundary  may  occur  in  intervals  of  one  unit  of  time. 

Example  4.1 

Consider  a network  with  three  state  variables  x^  and  x^.  Then 
the  sequences  are  (state  variables  enclosed  in  the  same  parentheses 
leave  the  boundary  simultaneously) : 


■f-2 

’^f-l 

"f 

"f-2 

"f-1 

"f 

"f-2 

"f-1 

"f 

^1 

*3 

’‘l 

’^2 

- 

’‘3 

(x^.x^) 

- 

(Xj^.Xj) 

*2 

- 

*2 

(x^,X^) 

- 

^3 

*3 

*2 

’‘l 

- 

*1 

(X2.X3) 

’‘l 

*2 

*2 

*3 

’'l 

- 

- 

(Xj^.X^.Xj) 

- 

(X2,X3) 

’‘l 

A typical  trajectory  appears  as  in  Figure  4.14. 

• Example  4 . 1 


The  complete  algorithm  consists  of  performing  all  of  the  possible 
backward  trajectories  as  in  Exeimple  4.1  and  constructing  the  associated 
non-break  feedback  control  regions.  In  order  to  describe  the  computa- 
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X 

4 


Figure  ' 


1 . 1.J 


F'/piL.'o’.  Uackwarci  Trajectory 


for  Example  4.1 
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tional  algorithm  we  will  follow  one  particular  optimal  trajectory  back- 
ward in  time  from  t^  for  a few  steps.  In  the  course  of  the  discussion 
we  will  demonstrate  the  method  by  which  the  leave-the-boundary  costates 
may  be  determined. 

Several  Steps  of  the  Computational  Algorithm  for  the  Construction 
of  the  Feedback  Space 

• Step  1 

• Operation  1 

(i)  Start  at  t^. 

(ii)  Pick  C 8^.  Then  J = L^-  See  Figure  4.15. 

(iii)  By  Corollary  3.2,  \ (t,)  =0  V-  x.  g L.. 

it  It 

(iv)  Arbitrarily  set  t^  - = 1. 

Xi^tf-i)  =1  V e Lf. 

<vi)  Solve  the  following  constrained  optimization  problem 
for  all  extremal  solutions; 

u*  = ARC  MIN  ^ A.(t.  ,)b^u 

ueu  ^ 


= ARC  MIN  ^ 

(Db^^fu) 

(4.87) 

U£U 

T 

subject  to  b.u  < -a. 

— 1 1 

V X.  E 
1 

'f-i 

(4.8B) 

V.T 

b.u  = -a , 
—1—  1 

V-  X . e 

1 

Vi 

(4.89) 

where  is  the  row  of  B,  Denote  thii'  solution 

—I  — 
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1 


set  have  used  the  notation  ^ instead 

of  the  usual  to  distinguish  this  non-breeik  optimal 
control  set  from  those  subsequently  obtained  in  this 
discussion.  By  Theorem  4.3  we  Joiow  that  is  globally 

optimal . 

• Operation  2 

(i)  Form  the  set  of  rays  ^ which  corresponds  to  ^ 
in  the  sense  of  Definition  (3.10). 

(ii)  Construct  the  convex  polyhedral  cone 

= Co(0  UV^_^)/0.  (4.90) 

,'j^^  is  a non-break  feedback  control  region  with 
associated  optimal  control  set  • 

At  this  point  we  prepare  for  Step  2 as  follows: 


Determination  of  the  Leave-the-Boundary  Costates 


We  shall  require  the  values  of  X,(t-  ,)  for  all  x.  C B . These 

1 f-1  1 f-1 

are  the  values  of  the  coefficients  of  the  hyperplane  H(t^  ^^)  whicli 
cause  it  to  lie  on  the  face  F(Y^_j^,  t^  of  .*/t  where 

corresponds  to  obtained  in  Step  1,  Operation  1,  part  (iv)  above. 

By  (4.77)  we  know  that 


A,  (t 


or 


\'Vi> 


¥ X,  e B 

1 f-r 


1 

I 


I 


In  order  to  determine  the  appropriate  value,  we  rotate  the  global 
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Hamiltonian  hyperplane  H(t^  about  the  hyperplane  i^*"f  i^  until 
it  touches  the  face  of  F(Y^  ^f  l'  while  holding  X(t^  con- 

stant for  all  e L^.  We  may  achieve  this  rotation  by  the  following 
algorithm.  Denote  by  x.  , x , x all  the  members  of  B . 


(i)  Set  k = 1. 

(ii)  Release  the  constraint  x^^  = 0 temporarily  by  adding  a 
slack  variable  ^ 0 to  the  appropriate  equation  of 
(4.89)  in  the  following  fashion: 

^u+2^=-a^  (4.91) 

The  variable  z,  is  now  a non-basic  decision  variable  of 
k 

the  linear  program  (4 . 87) - (4. 89) . By  (4.91)  we  have 

^ = -X,  (4.92) 

Note  that  Xj^  = 0 for  any  solution  to  (4 . 87)  - (4 , 89)  . 

(iii)  Adjoin  the  term 


to  t)ie  cost  function  (4.87).  The  new  linear  program  is 
now 


u*  = ARC  MIN  ^ (Db'^^u 

uelJ  x.cl^  , 

— '-1  f-1 

(4.94) 

T 

subject  to  b.u  < -a^ 

..  Xi  c Lf 

(4.95) 

u = -a  . 
—1—  1 

(4.96) 

(4,97) 


(iv)  Set  = 0. 

Therefore,  any  solution  to  (4 . 87) - (4. 89)  is  also  a 
solution  to  (4.94) -(4.97) . 

(v)  Start  with  any  solution  to  (4 . 87) - (4 . 89) . Holding  t^  ^ 

fixed,  solve  (4. 94)  - (4 .97)  parametrically  in  ^) 

for  increasing  linearly.  Stop  when  a change  in 

extreme  point  causes  to  enter  the  basis  with  a strictly 
positive  value  (a  degenerate  pivot  in  the  parametric  pro- 
gram may  cause  Zj^  to  enter  the  basis  with  zero  value)  . 

When  Zy^  is  strictly  positive  then  x^^  is  strictly  negative. 
We  have  therefore  caused  the  hyperplane  H(t^  ^)  to  lie  on 

^^^f-1'  ^f-1'  ' 

(vi)  Return  to  the  original  solution  of  (4 . 87) - (4 . 89)  chosen 
in  (v)  by  performing  reverse  pivots  from  those  executed 
in  the  parametric  program. 

(vii)  Set  k = k+1. 

(viii)  If  k > go  to  (x) . 

(ix)  Go  to  (ii)  . 

(x)  End. 


U{X>n  the  completion  of  the  above  algoritlim  we  have  caused  tlie 
global  Hamiltonian  hyperplane  H(t^  j^)  to  lie  on  F(y^_j^,  t^  x^^)  , 
F(V£_j^,  x^)  , . . . , f’(Y^  tj,  x^  ).  But  by  Theorem  4.1  we 

know  that  contains  the  single  mcmlier  F(Y^  , 
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) 

Therefore,  H(t^_^)  must  be  lying  upon  F(Y^  8^  and 

' ^2  ' ■’*'  must  be  the  appropriate  values  of 

the  leave-the-boundary  costates  at  t^ 


» Step  2 

• Operation  1 


(i) 

(ii) 


Start  at  boundary  junction  time  t^ 

Pick  C Then  1 = ‘-f  ^ See 

Figure  4.14. 


(iii) 


(iv) 

(V) 


(vi) 


^i<Vi’  = ^f  - tf-1  = 1 ^ 

~ (leave-the-boundary  value  determined  at  t^ 

''  ^ ^ h-r 

Arbitrarily  set  t^_^  - t^  ^ = 1 . 

X,(t^_2)=2  ^x.  eL^ 

A.  (t^_2>  = A. (t^_l)  + 1 vx.  eL^_^. 

Solve  the  following  constrained  optimization  problem 


for  all  extremal  solutions: 


11*  = ARC 

MIN  2^ 
ucU  x.eK  ^ 

(4.98) 

— 1 f-2 

subject  to 

T 

b . u < -a . 
— 1 1 

^ ’f-2 

(4.99) 

i-T 

b . u = -a . 
—1—  1 

* «i  ^ «f-2 

(4.100) 

Denote  this  solution  set  by  S2f_2-  By  Theorem  4.3  ^ 

is  globally  optimal. 
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• Operdtion  2 

(i)  Form  the  set  of  rays  2 which  corresponds  to 
in  the  sense  of  Definition  (3.10). 

(ii)  Construct  the  convex  polyhedral  cone 

,^^_2  = U V2’/'^f-l  (4.101) 

where  ^ is  given  by  (4.90).  Then  i^^_2  ^ non- 

brea)c  feedbac);  control  region  with  optimal  control  set 

f-2' 

• Determination  of  the  Leave- the-Boundary  Costates 
We  determine  the  leave- the-boundary  values  of 
e 6^  2 using  the  parametric  programming  scheme  detailed  following 
Step  (1). 

The  extension  of  the  cibove  steps  to  include  the  general  step  at 
boundary  junction  time  t^  is  a trivial  exercise  in  notation  and  shall 
not  be  presented  here.  All  of  the  linear  programming  manipulations 
called  for  in  the  algorithm  are  readily  performed  by  available  linear 
programming  packages  with  the  exception  of  t)ie  calculation  of  s'  1 
extremal  solutions.  A program  utilizing  Chernikova's  algorithm 
(described  in  Appendix  A)  was  devised  for  this  purpose. 

Examt:  xe  4.2 


A computer  t)rogram  was  devised  to  implement  the  above  algorithm 
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for  a fully  connected  five  node  single  destination  network  (a  total  of 
4 state  variables  and  16  control  variables) . For  this  situation,  com- 
plete execution  of  the  algorithm  calls  for  approximately  190  steps. 

All  operations  in  the  program  performed  with  acceptable  efficiency  (with 
regard  to  speed  of  computation  and  storage  requirements)  with  the 
notable  exception  of  the  program  which  calculates  all  of  the  extremal 
solutions,  x.ie  poor  efficiency  of  this  program  rendered  the  complete 
execution  of  the  algorithm  an  impracticality . •Example  4.2 

4. 5 Extension  to  General  Network  Problems 

In  this  section  we  indicate  how  the  geometrical  approach  used  in 
this  chapter  may  possibly  be  api:>lied  to  the  further  development  of  the 
backward  constructive  algorithm  for  problems  involving  multi-destination 
networks  with  arbitrary  weightings  in  the  cost  functional.  In  particu- 
lar, we  must  cope  with  the  four  complicating  properties  discussed  in 
Section  3.3.4.  We  present  in  Section  3. 3. 3. 2 the  geometrical  interpre- 
tation in  ^-space  of  three  of  these  problems  (global  optimality,  leave- 
the-boundary  costates,  subregions).  For  those  cases,  the  discussion 
centers  around  the  constraint  figure  , for  which  we  do  not  have  an 
explicit  representation.  In  particular,  we  talk  about  a particular 
class  of  faces  of  which  lie  adjacent  to  current  operating  points  - 

these  are  tlie  so-called  "L  -positive  faces." 

P 

Currently,  we  possess  no  method  for  obtaining  explicit  knowledge 
of  these  faces  for  the  general  multi-destination  network  problem.  Let 


i 

\ 


\ 


A 


-279- 


us  now  describe  our  dilenuna  in  more  specific  terms.  Assume  for  the 

sake  of  argument  that  the  current  set  of  operating  points  is  Then 

for  any  potential  boundary  junction  time  t^  e we  wish  to 

know  the  set  of  faces  , t , B ) ; actually,  we  require  the  hyper- 

P P P 

plane  in  which  each  of  these  faces  lies.  Now,  further  assume  that  Y 

P 

contains  an  extreme  point  of  fy,  call  it  Then  some  subset  of  the 

set  of  edges  of  emanating  from  ^ determine  the  required  hyperplanes. 
The  questions  now  becomes:  (i)  what  is  the  set  of  edges  of  y emanating 
from  y^  and  (ii)  which  edges  among  these  represent  the  required  hyper- 
planes? 

We  begin  our  examination  of  question  (i)  by  extending  to  general 
multi-destination  networks  the  description  of  edges  presented  in 
Section  4.2  for  single  destination  networks.  Once  again,  we  examine 
the  form  of  the  control  constraint  figure  D u < ^.  For  the  link  con- 
necting node  i to  node  k the  appropriate  constraint  equation  is 


u^  +u?  +. . .+u^,  ^+u^^^+. . .+u*^, +.  . .+u*?, +z.,  = C.,  (4.102) 

ik  ik  ik  ik  ik  ik  ik  ik 

z . , >0 
ik  — 

where  we  have  adjoined  the  non-negative  slack  variable  In  (4.102) 

tile  control  variable  u.,  is  termed  a direct  control  and  all  other  con- 

ik  

trol  variables  are  referred  to  as  indirect  controls  (Definition  4.3). 
Note  that  there  is  exactly  one  direct  control  per  link.  Now,  the  com- 
plete set  of  equations  of  ttie  type  (4.102)  corresponding  to  1'  u < C 


has  the  form: 


-280- 


where  we  have  eliminated  all  subscripts  except  for  ik  to  avoid  nota- 
tional  havoc.  Recall  that  r is  the  number  of  links  and  m is  the  dimen- 
sion of  the  control  vector.  Now,  notice  that  each  control  variable 
enters  into  exactly  one  equation  in  (4.103);  hence,  (4.103)  is  immedi- 
ately in  canonical  form  with  respect  to  any  set  of  r variables  chosen 
(either  u's  or  z's)  to  be  basic.  For  instance,  if  we  consider  the 
row  corresponding  to  C..  , then  we  pick  z or  some  control  u?,  (j^i) 

iK  IK  IK 

to  be  basic  and  set  its  value  equal  to  C . , and  the  value  of  the  re- 

ik 

maining  variables  to  zero.  We  do  this  for  each  row.  Note  that  since 
£ ^ 0,  every  basic  solution  to  (4.103)  automatically  has  u ^ 0. 

Therefore,  every  extreme  point  of  LI  is  represented  algebraically 
by  a point  whose  ik  coordinate  has  value  of  either  zero  or  Move- 

ment along  an  edge  of  U is  equivalent  to  picking  one  variable  which 

is  currently  basic  at  value  C (either  some  u?  , j^i,  or  z ) and  re- 

IK  IK  ik 

placing  it  in  the  basis  by  one  of  the  remaining  variables.  The  |X)ssibie 
points  which  may  occur  are: 
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1)  direct  control  exchanges  with  slack  on  link  ik 

2)  intermediate  control  exchanges  with  slack  on  link  ik 

3)  direct  control  exchanges  with  indirect  control  on  link  ik 

4)  indirect  control  exchanges  with  indirect  controls  on  link  ik. 

Notice  that  there  is  no  pivot  between  two  direct  controls  since  there  is 
exactly  one  direct  control  per  link.  Now,  the  edges  corresponding  to 
pivots  (1)  and  (2)  above  are  those  of  Type  1 and  Type  2 detailed  in  Sec- 
tion 4.2.  We  now  characterize  the  following  types  of  edges  of  :>/  emana- 
ting from  V corresponding  to  pivots  (3)  and  (4)  above: 


Type  3 - Exactly  tliree  coordinates  of  ^ change  from  their 
values  at  This  edges  corresponds  to  an  edge  of  U for 

which  a direct  control  and  an  indirect  control  change 
places  in  the  basis.  From  equations  (3.38)  and  (3.162)  we 
see  that  the  possibilities  are: 


= C., 
ik  ik 

'u?,  = 0 

ik 


pivot 


“ik  = 0 

= C., 
Ik  ik 


Ay?  = 


ik 


■ -‘'ik 


( k 

= 0 

1 

k 

1 

1 “ik 

pivot  , 

’ 

l“ik 

" ^ik 

(“ik 

= 0 1 

AyJ  = c 


ik 


Ay.  = -c., 

1 ik 


■«v.3  = 


'ik 


where  Ay^,  Ay?  and  Ay^  indicate  the  change  in  the  res(>ective 


coordinates  of  ^ when  moving  from  one  end  to  another  of  an 
edge. 


Type  4 - Exactly  four  coordinates  of  ^ change  from  their 
values  at  This  edge  corresponds  to  an  edge  of  U for 
which  two  indirect  controls  change  places  in  the  basis. 

From  equations  (3.38)  and  (3.162)  we  see  that  the  possibili- 
ties are: 
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We  therefore  have  a characterization  of  all  four  possible  types 
of  edges  of  Since  the  maximum  number  of  coordinates  of  which 

change  when  moving  along  any  given  edge  is  four,  the  constraint  figure 
A for  multidestination  networks  is  not  significantly  more  complex  than 
that  for  single  oestination  networks  (where  the  maximum  number  is  two) . 
One  should  be  able  to  exploit  this  fact  to  achieve  a better  under- 
standing of  the  nature  of  the  complicating  properties  of  the  algoritlim 
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for  general  multidestination  network  problems.  For  instance,  it  should 

not  be  difficult  to  establish  an  upper  bound  on  the  number  of  members 

of  t^,  1^) . This  is  equivalent  to  placing  an  upper  bound  on 

both  the  number  of  extreme  points  in  the  leave-the-boundary  costate 

set  and  the  number  of  subregions  in  R with  respect  to  L . However, 

P P 

in  order  to  implement  the  algorithm  one  must  possess  explicit  know- 
ledge of  the  set  of  edges  of  the  members  of  ,if(Y  , t , L ) which 

P P P 

emanate  from  A technique  must  therefore  be  devised  to  identify 

those  edges  of  U which  correspond  to  the  required  edges  of  ,*/. 


Chapter  5 


I 


CONCLUSION 


5. 1 Discussion 

In  this  section  we  attempt  to  place  in  perspective  the  accomplish- 
ments of  the  pi'eceding  chapters.  In  Chapter  1 we  express  the  desire  to 
develop  a d^  imic  procedure  for  message  routing  in  a store-and-forward 
data  communication  network.  As  an  initial  step  toward  achieving  this 
goal  a linear  dynamic  state  space  model  is  presented  in  Chapter  2 
which  represents  the  flow  and  storage  of  messages  in  such  a network. 
Although  the  model  is  sufficiently  general  to  represent  a stochastic 
user  demand  environment,  we  choose  to  consider  the  simplifying  case  of 
deterministic  inputs  for  initial  analytical  studies.  We  associate  with 
the  model  a linear  integral  cost  functional  which  is  the  weighted  total 
delay  experienced  by  all  the  messages  as  they  travel  through  the  network 
until  all  reach  their  respective  destinations.  The  desired  minimization 
of  the  cost  functional  results  in  a linear  optimal  control  problem  with 
linear  state  and  control  variable  inequality  constraints.  We  seek  a 
feedback  solution  to  this  problem. 

We  now  digress  a moment  to  discuss  the  oasic  approach.  Attempts 
to  apply  optimal  control  theory  to  problem  areas  in  which  otlier  tech- 
niques have  proven  inadequate  have  become  coinmon  practice  since  the 
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inception  of  the  field  in  about  1960.  Not  coincidentally , this  timing 
corresponds  to  the  beginning  of  the  era  of  computers,  a tool  which  is 
relied  upon  by  both  theorists  and  practitioners  of  optimal  control. 

Over  the  years  optimal  control  has  enjoyed  a number  of  successes,  most 
notably  in  the  areas  of  application  of  Linear-Quadratic-Gaussian  (Lqg) 
control  and  filtering.  However,  it  has  also  become  apparent  that 
"optimal"  in  the  mathematical  sense  does  not  always  imply  "best"  in 
some  more  general  sense,  so  that  optimal  control  theory  has  a great 
many  detractors.  See  for  example  Horowitz  [1976],  one  complaint  is 
that  in  minimizing  a single  objective,  optimal  solutions  often  ignore 
and  sometimes  aggravate  other  properties  of  the  system  performance. 
Also,  optimal  solutions  are  often  obtained  and  implemented  at  great 
computational  expense.  Finally,  if  the  objective  curve  is  relatively 
flat  near  the  optimum,  then  the  marginal  benefit  obtained  from  the 
optimal  solution  over  some  more  easily  obtained  sub-optimal  solution 
may  not  justify  the  additional  expense.  One  must  therefore  carefully 
weight  the  disadvantages  of  an  application  of  optimal  control  with 
respect  to  the  advantages.  The  fundamental  advantages  for  our  problem 
are  described  in  Sections  1.2  and  1.3.  We  shall  jx3int  out  some  dis- 
advantages in  the  course  of  this  discussion. 

Returning  now  to  the  optimal  message  routing  problem,  the  solution 
which  we  seek  calls  for  all  of  the  message  storage  in  the  network  to  go 
to  zero  at  the  final  time.  It  is  clear  that  this  requirement  does  not 
correspond  to  the  usual  data  network  mode  of  operation  in  which  tliere 
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is  a continuous  input  and  flow  of  messages  through  the  network.  How- 
ever, a particular  case  for  which  this  requirement  would  be  suitable 
is  when  one  desires  to  relieve  network  congestion  locally  in  time  by 
getting  rid  of  current  message  backlogs  stored  in  the  nodes.  In  this 
case,  the  inputs  to  the  networks  would  be  regulated  so  that  the  back- 
logs could  be  emptied  with  the  available  link  capacity.  Values  of 
constant  inputs  for  whicli  this  is  possible  are  given  in  Theorem  3.3. 

Having  formulated  the  optimal  control  problem,  the  remainder  of 
the  thesis  is  devoted  to  developing  a technique  for  finding  the  feed- 
back solution  for  the  case  in  which  the  inputs  are  constant  in  time. 

Of  course,  the  constant  input  assumption  further  restricts  the  appli- 
cability of  this  approach  to  real  network  problems.  However,  one  can 
conceive  of  situations  in  which  the  inputs  are  regulated  at  constant 
values,  such  as  the  backlog  emptying  procedure  described  above.  From 
the  optimal  control  viewpoint,  constant  inputs  appear  to  provide  us 
with  the  minimum  amount  of  structure  required  to  characterize  and 
construct  the  feedback  solution  with  reasonable  effort. 

The  backward  constructive  algorithm  is  described  in  principle  for 
multidestination  networks,  although  it  is  not  currently  understood 
how  this  algorithm  may  be  implemented  to  obtain  the  solution  for  this 
general  class  of  problems.  However,  the  complicating  aspects  of  the 
algorithm  disappear  when  we  consider  problems  involving  single  destina- 
tion networks  with  all  unity  weightings  in  the  cost  functional.  Our 
ability  to  completely  solve  only  this  special  class  of  problems  is  not 
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surprising  when  we  consider  chat  for  static  maximum  flow  problems  the 
general  multi- commodity  problem  remains  unsolved.  It  stands  to  reason 
that  a dynamic  multi-destination  problem  must  be  quite  difficultl 

Throughout  the  development  we  have  exploited  the  total  linearity 
of  the  problem  (in  dynamics,  cost  and  constraints)  to  obtain  many  of 
our  results.  In  particular,  we  have  frequently  employed  concepts  in 
linear  programming  both  in  the  synthesis  of  the  bac)sward  constructive 
technique  and  in  the  development  of  the  computational  algorithm.  It 
is  felt  that  this  approach  may  prove  helpful  in  other  optimal  control 
problems  with  a high  degree  of  linearity. 

Perhaps  the  primary  disadvantage  associated  with  the  feedback 
solution  is  the  amount  of  computer  storage  which  will  be  required  to 
implement  it.  The  feedback  control  regions  must  be  specified  by  a set 
of  linear  inequalities  which  in  general  may  be  extremely  large,  and  the 
optimal  controls  within  these  regions  must  also  be  specified.  On  the 
bright  side,  however,  we  note  that  if  the  particular  network  for  which 
we  are  providing  the  feedback  solution  contains  one  or  more  large  com- 
puters, the  required  storage  may  possibly  be  made  available.  Another 
disadvantage  is  the  amount  of  computation  required  in  the  original  con- 
struction of  the  feedback  solution.  In  particular,  the  program  devel- 
oped to  determine  all  the  extremum  solutions  to  a linear  program 
(Appendix  A)  requires  a substantial  amount  of  computation  time  and 
storage. 


1: 


5.2  Summary  of  Results  and  Related  Observations 


5.2.1  Results 

In  this  section  we  describe  briefly  the  primary  results  of  the 
thesis  in  the  order  in  which  they  appear. 

In  Section  3.2.1  we  develop  the  necessary  conditions  of  opti- 
mality associated  with  the  linear  optimal  control  problem  with  linear 
state  and  control  variable  inequality  constraints.  We  also  prove  the 
important  result  that  the  necessary  conditions  are  also  sufficient. 

Note  that  these  results  are  valid  for  general  deterministic  inputs. 

Starting  in  Section  3.2.2  we  assume  that  all  the  inputs  to  the 
network  are  constant  in  time.  In  the  same  section  we  state  and  prove 
a necessary  and  sufficient  condition  for  the  controllability  of  any 
initial  state  to  zero.  Utilizing  the  necessary  conditions  of  optimal- 
ity, we  prove  in  Section  3.2.3  that  the  feedback  regions  of  interest  are 
convex  polyhedral  cones  in  the  state  space. 

Section  3.3  is  devoted  to  a general  description  of  the  algorithm 
for  the  backward  construction  of  the  feedback  space.  The  steps  of  the 
algorithm  are  detailed  in  Section  3. 3. 2. 2.  In  Section  3.3. 3.1  we  show 
how  parametric  linear  programming  may  be  employed  to  efficiently  solve 
the  pointwise  linear  program  in  time.  The  geometrical  interpretation 
is  introduced  in  Section  3. 3. 3. 2 and  shown  to  be  useful  in  analyzing 
several  of  the  difficult  problems  associated  with  the  algorithm.  The 
construction  of  feedback  control  regions  is  detailed  in  Sections 
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3. 3. 3. 3 and  3. 3. 3.4  respectively. 

We  show  by  example  in  Section  3. 3.4.1  that  for  this  problem  there 
tnay  be  non-unique  costates  corresponding  to  state  variables  on  interior 
arcs. 

Chapter  4 is  devoted  to  problems  involving  single  destination 
networks  with  all  unity  weightings  in  the  cost  functional.  In  Section 

4.3  we  confront  those  properties  which  complicate  the  general  algorithm 
and  show  that  they  present  no  problem  for  the  class  of  problems  under 
discussion.  All  proofs  utilize  the  geometrical  interpretation  intro- 
duced in  Section  3. 3. 3. 2.  In  particular,  we  show  that  all  costates 
corresponding  to  state  variables  on  the  boundary  are  unique.  We  also 
prove  that  the  control  does  not  break  between  boundary  junction  times. 
In  Section  4.4  we  present  a complete  computer  implementable  algorithm 
for  the  construction  of  the  feedback  solution  for  this  special  case. 

5.2.2  Related  Observations 

Although  the  approach  which  we  have  taken  may  be  limited  in  prac- 
tical applicability,  the  results  which  we  have  obtained  can  provide 
valuable  insight  into  the  dynamic  message  routing  problem.  In  this 
section  we  discuss  several  issues  from  this  point  of  view. 

We  begin  with  a discussion  of  the  state  information  required  to 
implement  the  optimal  control.  As  pointed  out  in  Section  2.5,  by  spe- 
cifying a feedback  control  function  of  the  type  £(t)  = u(t,  J^(t) ) we 
are  assuming  that  total  information  regarding  the  storage  state  x(t) 
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is  available  at  each  node  of  the  network  for  control  decisions  at  that 
node.  Such  a scheme  requires  a centralized  method  of  observing  and 
communicating  the  state  of  the  system  at  every  point  in  time.  However, 
in  practice  one  may  wish  to  consider  control  strategies  which  do  not 
require  total  knowledge  of  the  state  at  every  time.  For  instance,  it 
may  be  desirable  to  have  a decentralized  strategy  that  makes  control 
decisions  ba^od  on  local  state  information  only. 

Such  considerations  lead  us  to  examine  the  nature  of  the  state 
information  requirements  associated  with  the  solution  we  have  obtained 
in  the  specific  case  of  constant  inputs.  We  have  found  that  the  struc- 
ture of  the  solution  is  such  that  regions  in  the  state  space  of  common 
optimal  control  are  convex  polyhedral  cones.  This  situation  is  depicted 
conceptually  for  a two  dimensional  example  in  Figure  5.1. 
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We  see  from  Figure  5,1  that  the  optimal  control  is  always  a func- 
tion of  both  and  X2.  A completely  decentralized  scheme  is  therefore 
not  possible.  However,  due  to  the  nature  of  the  optimal  control  re- 
gions, we  do  not  require  exact  values  of  x^  and  x^  but  only  their 
relative  values.  For  example,  we  need  only  know  that  — > a,  and 

Xj^  - 1 

> 0 to  conclude  that  u*  is  the  appropriate  optimal  contrcl.  This 

relationship  may  hold  true  for  relatively  long  periods  of  time,  so 

that  periodic  rather  than  continuous  updating  of  the  ratid  — should 

^1 

prove  to  be  rather  good.  In  fact,  based  upon  the  current  state  and 
optimal  control  it  is  easy  to  predict  (due  to  linearity)  the  time  at 
which  the  state  will  move  from  the  current  feedback  control  region 
into  an  adjacent  feedback  control  region  in  the  absence  of  outside  dis- 
turbances or  variations  in  the  system  parameters.  Therefore,  one  may 
wish  to  update  the  state  information  (and  the  optimal  control,  if 
called  for)  at  these  predicted  switch  times . Although  this  scheme  would 
not  cut  down  on  the  amount  of  state  information  required,  it  drasti- 
cally reduces  the  frequency  with  which  it  must  be  supplied. 

For  the  purpose  of  our  next  discussion  we  refer  to  the  network 
problem  of  Example  3.1.  From  Figure  3.9  we  see  that  if  the  state  lies 
in  the  region  between  the  lines  X2~3x^  = 0 and  (not  inclu- 

ding these  lines)  then  any  admissible  values  of  the  control  variables 
u^2  ^^21  optimal.  Therefore,  looping  of  messages  between 

nodes  1 and  2 of  Figure  3.7  may  be  optimal.  However,  such  looping  is 


not  desirable  in  data  communication  networks  for  obvious  reasons.  This 
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problem  is  an  example  of  peripheral  considerations  which  may  be  ignored 
by  optimal  solutions  with  limited  objectives.  In  order  to  alleviate 

this  situation,  one  must  modify  the  problem  statement  appropriately.  ! 

■j 

I 

For  this  case,  optimal  solutions  may  tend  to  avoid  looping  if  we  incor- 

T 

porate  a term  linear  in  the  control,  say  it  u,  into  the  cost  functional. 

The  idea  here  is  to  discriminate  against  frivolous  use  of  the  links.  | 

i 

Fortunately,  _his  modification  does  not  complicate  the  solution  tech- 
nique presented  in  this  thesis  since  the  only  change  from  before  is 
that  the  pointwise  minimization  is  now 

U*(T)  = ARG  MIN  [(^(T)B  + IT ) ”^U  ] . 

ueU  I 

- 

5.3  Contributions 

In  this  thesis  we  expand  the  areas  of  application  of  optimal  con-  ■" 

trol  theory  to  include  the  important  class  of  problems  involving  message 
routing  in  data  communication  networks.  As  in  many  previous  applica- 
tions, we  find  that  this  approach  may  provide  significant  advantages  ^ 

over  previous  techniques  used  to  attack  the  problem  at  hand.  In  this 
case,  we  obtain  the  dynamic  and  closed- loop  properties  lacking  in  the 
queueing  theory  approach.  The  concept  of  applying  optimal  control 
theory  to  the  dynamic  routing  problem  was  pioneered  by  Segal 1 [1976] 
and  this  is  the  first  work  to  provide  a detailed  solution  to  the  optimal 
control  problem.  It  is  hoped  that  the  success  of  this  effort  will 


stimulate  furtlier  investigation  into  this  approach  so  tJiat  its  full 
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potential  may  be  realized  and  understood  with  respect  to  other  tech- 
niques . 

From  the  control  theory  point  of  view,  it  is  commonly  known  that 
optimal  feedback  solutions  are  in  general  very  difficult  to  determine. 
This  is  particularly  true  for  problems  involving  state  variable  inequal- 
ity constraints,  where  one  can  at  best  hope  to  find  such  solutions  only 
for  special  problems.  We  have  found  a complete  feedback  solution  to 
a meaningful  linear  optimal  control  problem  with  linear  state  and  con- 
trol variable  constraints  and  consider  this  achievement  to  be  a signi- 
ficant contribution  to  the  theory  of  deterministic  optimal  control.  Of 
particular  interest  is  our  novel  combination  of  necessary  conditions, 
dynamic  programming  and  linear  programming.  In  addition,  we  have 
developed  a theory  based  on  geometrical  concepts  in  linear  programming 
for  the  analysis  of  linear  optimal  control  problems  with  linear  con- 
straints. 

5.4  Suggestions  for  Further  Work 

We  view  this  work  as  an  initial  investigation  into  the  feasibility 
of  applying  optimal  control  theory  to  problems  of  dynamic  routing  in 
data  communication  networks.  As  such,  we  realize  that  many  more  ques- 
tions remain  unanswered  than  have  been  satisfactorily  resolved. 

The  primary  challenge  is  to  gain  an  understanding  of  the  method  by 
which  the  backward  constructive  algorithm  for  general  network  problems 


may  be  implemented  on  the  computer.  From  our  arguments  in  Chapter  4 it 
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is  clear  that  the  geometrical  interpretation  is  a powerful  tool  in 
understanding  the  various  properties  associated  with  the  algorithm. 
Although  we  know  that  the  special  properties  of  Chapter  4 do  not  hold 
in  general,  we  feel  that  the  geometrical  interpretation  may  be  used  to 
identify  such  properties  as  do  hold.  This  possibility  is  discussed  in 
faction  4.5.  Hopefully  these  properties  may  be  adapted  to  a computer 
implementable  algorithm. 

An  important  area  for  significant  advance  is  in  the  development  of 
an  efficient  algorithm  which  computes  all  of  the  extreme  point  solutions 
to  a linear  program.  As  mentioned  in  Section  3. 3. 3.1  this  is  a classi- 
cal problem  in  linear  programming  and  to  the  author's  knowledge  there 
is  no  algorithm  available  which  is  efficient  enough  for  our  purposes. 
Perhaps  a good  place  to  start  is  by  attempting  to  exploit  the  special 
structure  of  single  destination  networks  to  construct  an  efficient 
algorithm  for  that  case.  At  any  rate,  the  desirability  of  the  backward 
constructive  technique  will  be  greatly  enhanced  upon  finding  a satis- 
factory solution  to  this  problem. 

Finally,  of  course,  there  remains  to  extend  the  feedback  solution 
first  to  general  deterministic  inputs  and  ultimately  to  the  most  real- 
istic case  of  stochastic  inputs.  It  is  clear  that  both  of  these  prob- 
lems are  extremely  difficult  due  to  the  state  variable  inequality  con- 
straints. Segall  [1976]  presents  a sufficient  condition  of  optimality 
in  the  stochastic  case  when  the  inputs  are  modelled  as  mutually  inde- 
pendent jump  processes  witli  positive  jumps  only.  However,  the  condition 


is  expressed  in  terms  of  the  partial  differential  equation  of  the  opti- 
mal cost  to  go  and  solution  by  numerical  integration  is  out  of  the 
question  due  to  excessively  high  dimensionality.  Therefore,  alternate 
solutions  must  be  investigated. 


Appendix  A 


FINDING  ALL  EXTREME  POINT  SOLUTIONS 


TO  THE  CONSTRAINED  OPTIMIZATION  PROBLEM 


Operation  1,  part  (a) , of  tlie  backward  constructive  algorithm 
calls  for  finding  all  of  the  extreme  point  solutions  to  the  constrained 
optimization  problem.  At  a fixed  instant  of  time  this  problem  is  the 
linear  program  (the  time  variable  has  been  dropped) : 


where 


subject  to 


z = y j X?b?u 

1 p-1 


ID  U ^ C 

b?u  = -a? 
—1—  1 

u > 0 


V-  X?  e 8 - 

1 p-1 


where  b?  is  the  row  of  B corresponding  to  x^ . Suppose  we  have  utilized 
a standard  linear  program  routine  employing  the  simplex  algorithm  to 


find  a single  extreme  point  solution  to  (A.1)-(A.5).  Call  this  solu- 
tion u*.  Then  if  we  denote  z*  = A?b^u*,  the  hyperplane 

xhl  , 

1 p-1 
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•z*  = 

is  tcingent  to  the  convex  polyhedron  U'  at  u*  and  every  point  contained 
in  the  face  of  tangency  is  a solution  to  (A.1)-(A.5).  Consequently,  in 
order  to  find  all  extreme  point  solutions  to  (A.1)-(A.5)  we  must  find 
all  vertices  of  the  convex  polyhedron: 


el 


aWu 

i-a— 


p-1 


(A.  6) 


U' 
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z* 


(A. 7) 


Rubin  [1975]  demonstrates  that  the  problem  of  finding  all  the  ver- 
tices of  an  m-dimensional  convex  polyhedron  can  easily  be  formulated 
as  the  problem  of  finding  all  the  edges  of  an  m+l-dimensional  convex 
polyhedral  cone.  Furthermore,  Chernikova  [1964] , [1965]  has  presented 
algorithms  for  finding  all  the  edges  of  a convex  polyhedral  cone  repre- 
sented by  a set  of  homogeneous  linear  equations  and  inequalities. 

Bloom  [1976]  has  programmed  Chernikova 's  algorithm  as  a FORTRAN  sub- 
routine and  reports  on  computational  experience  gained  using  it.  He 
finds  that  for  problems  of  reasonably  large  size  the  amount  of  computa- 
tion and  memory  required  to  execute  the  algorithm  is  excessive.  At  the 
root  of  the  problem  is  the  extreme  numerical  sensitivity  of  the 
algorithm  to  the  value  of  z*.  If  z*  is  slightly  too  large,  then  the 
hyperplane  (A. 6)  does  not  intersect  U’ , and  hence  LI"  is  empty.  If  z* 


I 

1 
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i 

i 


is  slightly  too  small,  the  hyperplane  (A. 6)  intersects  U*  below  the 
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EXAMPLE  OF  NON-BREAK  AND  BREAK  FEEDBACK  CONTROL  REGIONS 


We  present  here  a simple  example  to  illustrate  the  two  types  of 
feedback  control  regions.  The  network  is  shown  in  Figure  B-1  and  the 
cost  functional  is  taken  to  be 


•'t„ 


X23(t))dt. 


The  ^-constraint  figure  for  this  network  is  presented  in  Figure 
B-2.  We  shall  be  interested  in  the  operating  points  @ - (?)  , and 
the  coordinates  are  indicated  in  the  order  V-^2'  ^32  ^23’ 

controls  corresponding  to  these  points  are  listed  in  Figure  B-3.  The 
particular  backward  optimal  trajectory  which  is  depicted  in  Figure  B-4 
has  x^2  leaving  boundary  first  backward  in  time,  x^2  leaving  next  back- 
ward in  time  and  finally  x^2  leaves  the  boundary.  The  regions  corres- 
ponding to  controls  (T)  , and  (4)  are  non-break  feedback  control 
regions.  The  region  corresponding  to  control  (T)  (below  the  breakwall 
Xj^2  ” ^^^32  ~ ^ ^23  “ is  a break  feedback  control  region. 
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